The Random-Max Percentile: Grading Every Backtest Against Its Search's Random Maximum

Key Takeaways

• RMP (Random-Max Percentile) is a new per-row column in our scan results and paid-plan exports: the probability that the best result pure chance could produce, across the N trials your scan actually ran, lands below this row. An RMP of 0.97 reads: even the luck record of a search this size sits below this row with 97% probability.
• It exists because deflated Sharpe has a blind zone. Across one 5.81-million-trial search in our own audit, the best DSR among rows with 1,000 to 2,999 trades was 0.001, and at 3,000 trades or more it was 0.000, real edge or not. All eight populations we diagnosed saturate the same way, because DSR's hurdle is set in Sharpe units and per-trade Sharpe shrinks as frequency grows.
• The mathematics is deliberately old: one minus RMP is the Šidák multiple-testing correction, published in 1967. The operational part is new: a scanner records its own trial count, so the correction computes per row, in closed form, at hundred-million-backtest scale. RMP tracks PSR's ordering (rank correlations agree to three decimals across our audit); what changes is the hurdle.
• In a synthetic test that plants the same-strength edge at five trade frequencies, DSR's detection of a strong edge (whole-period t near 6) falls from 100% at 30 trades to 0% at 3,000 while RMP stays at or above 94%; a moderate edge collapses the same way at lower levels. On the real corpus, passing is rare: among the 101 million audited rows with at least 30 trades, 780 cleared 0.95, about 770 per 100 million backtests. It is an exploratory gauge, not a certification.

1. What RMP is

Scan results on our scanner carry a new statistics column: RMP. It sits next to the deflated Sharpe column (DSR) in the results table, completes the PSR-DSR-RMP trio in each row's detail (PSR, the Probabilistic Sharpe Ratio, is the single-bet significance gauge), and is written into the export; like DSR, the numbers are part of paid plans. The definition fits in one sentence:

RMP (Random-Max Percentile): the probability that the best result pure chance could produce, across the N trials your scan actually ran, lands below this row.

An RMP of 0.97 reads: even the luck record of a search this size, the single best score a zero-edge version of your entire scan would post, sits below this row with 97% probability. An RMP near zero reads: this row does not stand out from what luck alone would have produced somewhere in your search. Most rows of any real search read near zero, and that is the correct answer.

The formula is one line:

The whole gauge. q is the row's evidence score: per-trade Sharpe scaled up by the square root of the trade count, corrected for the shape of the returns (skew and kurtosis are the row's own sample values). Phi(q) is the chance that one zero-edge trial stays below it, and the power N asks all of the scan's recorded trials to. Source: the production column's definition; footnote 2 for the PSR relationship.

RMP = Φ(q)^N, where q is the row's t-statistic, adjusted for skew and fat tails the way PSR does it,¹ and N is the number of trials your scan ran. Φ(q) is the probability that one random draw stays below q; raising it to the Nth power asks all N of them to. Because Φ(q) is monotone in q, RMP orders strategies very nearly as PSR does; across the 24 files of our audit the two orderings agree to three decimal places of rank correlation (identical by construction for rows with the same trade count; the three-decimals figure is the check across different trade counts). What it changes is the hurdle: from "better than chance once" to "better than the best of everything this search tried."

The rest of this article is the paper trail: why the column exists, how it works, how we stress-tested it, and what it can and cannot tell you.

2. The blind zone it fills

The Deflated Sharpe Ratio comes from Bailey and López de Prado.² It answers a sharp and honest question: you searched N variations and kept the best-looking one, so is that row's Sharpe above what the best of N lucky tries would show? It estimates the expected maximum Sharpe a zero-edge search of your size would produce (our exports record it as SR*), and grades your row against that hurdle. We lean on DSR constantly: it is the statistic behind the luck bar that filtered 100 million BTC backtests down to 3,130, and it carried our study of what happens when you run the same search twice. Nothing in this article retires it.

But the hurdle is set in Sharpe units, and that choice has a consequence. The statistical evidence in a track record is not its Sharpe; to a good approximation it is per-trade Sharpe multiplied by the square root of the trade count, the familiar t-statistic. (Results screens on our scanner show annualized Sharpe, per-trade Sharpe times the square root of 252 by convention; every deflation number in this article runs per-trade.) Two strategies with the same whole-period evidence, one trading 30 times and one trading 3,000 times, differ in per-trade Sharpe by a factor of ten, by construction, and neither carries more proof than the other. Hold a bar fixed in Sharpe units and raise the trade count: in evidence units, that bar is a hurdle growing with the square root of frequency. Busy strategies are not failing the test. They are being asked for a different, much larger quantity of proof, purely because they trade often.

Mixed populations, which is what real scans are, make it worse. The expected-maximum hurdle is computed from the spread of Sharpes across the whole search, and rows with few trades have the noisiest, widest-spread Sharpes. They set the bar. In synthetic populations built from a real scan's trade-count distribution, that bar averaged 0.18 per-trade Sharpe, while a genuinely strong strategy trading 3,000 times has a per-trade Sharpe of about 0.11. Its evidence is overwhelming. Its Sharpe can never reach a bar set by the fidgety end of the population.

The audit numbers behave exactly as the geometry predicts. Across 24 of our own scan exports, 116 million rows: in one 5.81-million-trial search, the best DSR among rows with 1,000 to 2,999 trades was 0.001 (N = 5.81M), and at 3,000 trades or more the best was 0.000. Read the same two buckets with the new column and the picture inverts: their strongest rows come out at RMP 0.999 and 1.000. All eight populations we diagnosed (three coin tiers, long and short, plus two single-coin control runs) saturate the same way. DSR is not broken. It answers a different question, in units that stop being informative where trade counts are high. What was missing is a gauge for that region, and that region is exactly where high-frequency strategies live.

3. How it works

Picture a coin-flipping contest. A thousand people each flip ten times, and the winner shows nine heads. Impressive? Not really: the best of a thousand zero-skill contestants is expected to look that good. Whatever score you personally posted, the only fair question is whether it beats what the field's champion gets for free.

A parameter scan is that contest, and the scanner knows exactly how many contestants entered, because it ran them. RMP asks the contest question directly. If one zero-edge trial stays below your row's adjusted evidence score q with probability Φ(q), then all N of them do with probability Φ(q)^N, and that is the whole formula: the probability that your row tops the luck record of a search your size.

Statisticians will recognize the construction. One minus RMP is the Šidák multiple-testing correction, published in 1967;¹ the bar itself is decades old and thoroughly conventional. Related machinery exists across the literature: bootstrap reality checks grade a whole search's best result by resampling its return series, and multiple-testing haircuts adjust t-statistic thresholds when the number of trials must be guessed. What is new here is operational, and it is the part a scanner is uniquely positioned to do: the search records its own N, every row already carries the t-statistic and the shape terms that adjust it, so the correction computes per row, in closed form, with no bootstrap, across a hundred million backtests. In our earlier articles the luck bar was a single pass-or-fail line for a whole search. RMP is the same idea turned into a continuous per-row gauge, in evidence units rather than Sharpe units, which is precisely why it does not saturate where DSR does.

Two rows through the formula. To make it concrete, push two illustrative rows through one of those synthetic searches: N = 20,000 trials, Sharpe-unit bar 0.18, one consistent world. Round numbers, with the shape adjustment set aside for the walkthrough (the live column applies it):

	Row A: the busy one	Row B: the flashy one
Trades	3,600	30
Per-trade Sharpe	0.10	0.50
Evidence, q ≈ Sharpe × √trades	6.0	2.7
PSR: better than chance, alone?	≈ 1.000	0.996
DSR: above the expected best, in Sharpe units? (bar 0.18)	0.000	0.958
RMP: above the luck record of all N?	0.99998	≈ 0

Row A is this article's protagonist. Its per-trade Sharpe, 0.10, sits far below the Sharpe-unit bar, so DSR reads 0.000. But its evidence sits six standard errors out, the hurdle that beats the luck record of 20,000 zero-edge trials sits at about 4.6, and Φ(6.0) raised to the 20,000th power comes out at 0.99998. The row stands above its search's record.

Row B is the mirror image, and the more instructive one. Thirty trades at a hefty per-trade Sharpe look superb alone (PSR 0.996) and clear the expected-best bar too (DSR 0.958). But an evidence score of 2.7 is a value this search produces by pure chance dozens of times over, so against the full luck record RMP rounds to zero. The two columns disagreeing is not noise. It is the size of the search making itself visible in one row and staying invisible in the other.

4. Calibration: does the bar hold?

A closed-form bar is only as good as its approximations, so before shipping we stress-tested them against simulated null worlds where the truth is zero by construction.

On trade returns with the take-profit and stop-loss geometry our backtests actually produce, RMP's false-alarm rate, measured as the chance that a whole zero-edge search lights up even one row, runs 3.5% to 5.8% at 300 or more trades against a 5% design target. Below 300 trades it under-fires: bounded trade returns make the gauge conservative rather than trigger-happy at low counts. One shape stays stubborn: strategies with unbounded, purely one-sided returns (lottery-like payoffs with no stop) trip the bar at 1.5 to 2 times the nominal rate even at search sizes in the millions. Below roughly 100 trades the approximation is shape-sensitive in both directions, so treat very-low-trade RMP values as soft.

The independence assumption gets its own audit. RMP counts every trial as its own contestant, but neighboring parameter settings produce overlapping, correlated strategies, so a scan's effective field is smaller than its N. That inflates the hurdle, which errs in the safe direction: fewer false lights. And when we checked the closed-form bar against block-resampled reshuffles of a real scan at its real N, it sat in the middle of the pack of resampled luck records rather than off to one side; the percentile-level version of that audit belongs to the longer write-up this column comes from. The cheap formula tracks the expensive resampling answer at the scale where the expensive answer is unaffordable to compute per row.

5. Power: same edge, five frequencies

A calibrated gauge can still be a blind one, so we tested detection the adversarial way: plant edges of known strength, vary only the trade count, and see which gauge finds them. The null populations were 20 searches of 20,000 strategies each, zero true edge, trade counts drawn from a real scan's distribution (median about 3,600 trades). Into copies of that world we planted strategies whose whole-period evidence is fixed while frequency varies.

Detection rates for a strong edge (whole-period t near 6) and a moderate one (t near 4):

Trades	DSR > 0.95, strong	RMP > 0.95, strong	DSR > 0.95, moderate	RMP > 0.95, moderate
30	100%	100%	99%	57%
100	100%	99.5%	77%	42%
300	94%	97%	23%	35%
1,000	10%	94%	0.1%	32%
3,000	0%	94%	0%	29%

Read the strong column first. The same edge, carrying the same evidence, goes from certain detection to invisible as DSR's Sharpe-unit hurdle climbs past it, and at 3,000 trades detection is zero. RMP stays at or above 94% across the entire row, because its hurdle lives in evidence units and the evidence never changed. That flat line is the whole point of the column.

Read the low-trade rows just as carefully, because they cut the other way. At 30 trades with a moderate edge, DSR detects 99% and RMP only 57%. RMP's hurdle for a 20,000-trial search sits at about 4.6 standard errors, which is brutal, and at low trade counts DSR is simply the sharper instrument. This is why the product ships both. Complementary gauges, not a replacement.

One more honest read of the same table: the RMP columns drift down before they level off, and that drift is the estimator, not the hurdle. At 30 trades the t-statistic is itself a noisy, skewed estimate, and when the bar sits above the true signal, noise crosses it more often than it misses. As trade counts grow the estimate settles and detection converges to its steady level; the moderate column's 29% at 3,000 trades is within a point of what the formula predicts for an ideal estimate, 28.8%. The hurdle never moved.

Two footnotes to the same experiment. On the pure-null populations, out of 400,000 zero-edge strategies, DSR lit up 18 times and RMP twice: both hurdles hold their false alarms to a handful. Note that this asymmetry runs against the new column: in the power table RMP is competing with a DSR that spends roughly nine times its false-alarm budget on the same null, and the high-frequency collapse happens anyway. And PSR, the single-bet gauge, detected the planted edges almost without fail while also lighting up 19,885 null strategies, almost exactly 5% of 400,000. PSR is doing its job, which was never search-aware. Anyone screening a large scan on PSR alone is reading thousands of false positives as discoveries.

6. In the product

The column is live: next to DSR in the results table, in the PSR-DSR-RMP trio of each row's detail, and as a column in paid-plan exports. Three habits make it useful.

Expect near-zero, mostly. A small search we ran while testing (20 trials on one coin) put every row's RMP near zero, and that was the correct answer: nothing in it beat even a 20-contestant luck record. On million-trial searches the pass rate is single digits per million: among the 101 million audited rows with at least 30 trades, 780 cleared 0.95, about 770 per 100 million backtests.

Read 0.95 as one specific sentence. The bar is set so that a pure-luck search of your size has only a 5% chance of putting even its single best row past it. A row above it stands above the luck record of your own search, under the stated assumptions, on the tested window. Rows at 0.6 or 0.8 are not failures; they are measured distances from the bar, which is exactly the information a pass/fail line throws away.

Use the three gauges as three questions. PSR: is this row better than chance, taken alone? DSR: is it above the expected best of my search, the sharpest question at low trade counts? RMP: is it above the search's full luck record, the question that stays answerable at 3,000 trades? A busy row with DSR 0.000 and a high RMP is not a contradiction. It is a strategy whose evidence is strong but whose per-trade Sharpe lives below a bar set in units that stop being informative at its frequency.

Do the rare passes mean anything forward? In the audit search above, the rows RMP passes carried walk-forward profiles on par with the rows DSR passes in the regime where DSR still speaks: median walk-forward efficiency 0.57 across the 665 rows RMP passes, against 0.56 for the 16 rows DSR passes there (walk-forward efficiency compares out-of-sample performance to in-sample; 1.0 is full carry-over). The test that would settle it properly, a pre-registered fresh scan graded forward, is the natural next study.

And if this would be your first scan rather than your next one, the starting point is not a blank grid: scan the neighborhood of a strategy you already trust, the parameter ranges you would have hand-tuned anyway, and read the three gauges on its best row against the luck record that search just set. (That record covers the trials that search ran; Section 7 is about how the bar should grow across searches.)

Every paid-plan export carries the inputs behind the column (its t-statistic, per-trade Sharpe, skew, tail weight, and the search's trial count), so the number is reproducible from the file alone.³

7. Limitations

RMP assumes independent trials; real scans overlap, which pushes the gauge conservative. Within each row it also treats the trades themselves as independent draws, the same assumption PSR and DSR already make; the block-resampled audit, which preserves the real data's time structure, is the standing check on both. It assumes a normal approximation for the adjusted t-statistic; that holds well for bounded trade returns at 300-plus trades, under-fires below, and over-fires by 1.5 to 2 times on unbounded lottery-shaped returns. It inherits everything a backtest already assumes about fills and fees. It counts this scan's trials only: rerun and redesign searches across weeks and your true trial count grows past what any single export records, with a bar that should grow with it. And it is a statement about one search's luck on one window of one market's history: an RMP above 0.95 is not a probability of profit, and the label we use internally is "exploratory."

Frequently asked questions

Is RMP a probability of making money? No. It is the probability that the best result pure chance could produce, in a search the size of yours, lands below this row. A high RMP says the number is hard to explain as search luck on this window. It says nothing about next month, position sizing, fees you did not model, or regimes you did not test.

Does RMP replace DSR or PSR? No, and the synthetic table shows why not. At 30 trades a moderate edge trips DSR 99% of the time and RMP only 57%: on small, low-frequency searches DSR remains the sharper gauge. RMP takes over where DSR saturates, which in our audit is roughly the region past a thousand trades (in the synthetic table the detection crossover comes earlier, around 300). PSR stays what it always was, the single-bet question, useful precisely because it ignores the search.

Doesn't this undercut your earlier luck-bar articles? No, and the distinction matters. Those searches were graded with DSR inside the regime where it works, and their counts stand. What this article adds is that the same bar could not have seen a high-frequency edge had one been there: the earlier claims are about the rows the bar did pass, not about the region it cannot read. That region now has its own gauge.

Why is every RMP on my scan near zero? Because most rows of a real search do not beat that search's luck record, and the column does not pretend otherwise. Check the trial count in your export: the larger the search, the higher the record. If your best rows sit at 0.6 or 0.8, that is a measured distance from the bar, not a failure.

What assumptions am I accepting when I read it? The ones Section 7 lists: independent trials, a normal approximation for the adjusted t-statistic, and everything a backtest already assumes about fills and fees. The practical translation: expect it conservative on overlapping grids and at low trade counts, and 1.5 to 2 times too eager on stop-less, lottery-shaped return profiles. The gauge is exploratory by design.

---

Auto-trading and trading carry a risk of losing your principal. This article is educational, does not guarantee profit, and past backtest results do not predict future returns.

Footnotes

1. Šidák (1967), "Rectangular Confidence Regions for the Means of Multivariate Normal Distributions," Journal of the American Statistical Association, 62(318). The q inside RMP = Φ(q)^N is the row's per-trade t-statistic divided by the same skew-and-kurtosis adjustment PSR uses (per-trade Sharpe columns are capped at ±6.2994; no number quoted here sits on a cap), and one minus RMP equals the Šidák-adjusted one-sided p-value of that statistic. RMP > 0.95 is equivalent to q clearing Φ⁻¹(0.95^(1/N)): about 4.6 standard errors for a 20,000-trial search, about 5.6 for a 5.81-million-trial one. ↩ ↩²

2. Bailey & López de Prado (2014), "The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality", The Journal of Portfolio Management, 40(5). Our exports record each population's trial count and its expected-maximum bar (SR*) in the file metadata, and DSR figures in this article always quote the trial count they were computed against. ↩

3. The synthetic study: null populations of 20,000 zero-edge strategies per search, trade counts resampled from a real scan's distribution, trade returns following an 80/20 mixture of a two-point take-profit/stop-loss payoff and Gaussian noise, 20 independent searches, fixed seed; planted rows fix whole-period evidence while varying trade count. Detection is the fraction of 2,000 replicates crossing 0.95. The audit figures inherit the modeling caveats of the underlying scans as published with those articles (taker fees, flat slippage, no perpetual funding on short legs); pass counts are taken among rows with at least 30 trades and full validation columns, the audit's eligibility filter. The audit corpus is our own scan exports; the saturation buckets quote the search they come from (N = 5.81M). ↩