How Many Backtests Is Too Many? We Ran the Same Search Twice to Find Out

Key Takeaways

• A single strategy, its trades bit-for-bit identical, grades as luck in one scan (deflated Sharpe 0.035, inside a 21.9M-trial search) and close to real in another (0.919, inside 14.8M). The strategy never changed; the crowd it was measured against did.
• We ran one 5.7-year BTC and ETH perpetuals search twice, changing one thing: run B removed the swept configurations of two rarely-firing indicators (4 of the 20 indicator-timeframe slots). The pass bar that luck alone sets (SR*) fell from 0.855 to 0.397 per-trade (long) and from 1.165 to 0.378 (short); among strategies with 30 or more trades, run A cleared zero rows and run B cleared 3,798 (long) and 840 (short).
• The bar moved through variance, not count: dropping 28% of trials moved the count term by about 1 percent, while the variance term fell 2.1x (long) and 3.0x (short).
• The winners are concrete: the best long made +187% over the 5.7 years at a 72% win rate (DSR 0.925 against 14.8M trials), and the best short +53% at 83% (DSR 0.919).
• Run B's headline numbers were computed from run A's export alone, before run B ran: the trial counts landed to the digit, the luck bar to seven decimals. Every number in this article comes from the two scans' own exports: per-row deflated Sharpe and trade count, plus each population's trial count, variance, and SR*.

1. How many backtests is too many?

It is the wrong worry. The penalty for searching hard grows with the logarithm of how many backtests you run, and logarithms are stubborn: our own experiment dropped 28 to 33 percent of its trials, direction by direction, and that term moved by about 1 percent. What moved the bar itself, by 2.15x in one direction and 3.08x in the other, was not how many strategies we tested but which kinds of strategies were allowed into the pool. Four indicator slots (a slot is one indicator on one timeframe) that rarely fired, flooding the pool with results built on a handful of trades, inflated the population's variance so much that every other strategy in the scan was graded against a hurdle more than twice as high.

That claim is easy to assert and hard to trust, because scans differ in a hundred ways. So we built the cleanest test we could: the same search, run twice, with one dial changed. By the end there are three concrete strategies on the table, entry configurations, win rates and returns included, and next to each of them a number that says exactly how far to trust it.

2. What sets the bar: SR*, the deflated Sharpe hurdle

Two definitions carry this whole article.

SR* is the best Sharpe ratio you would expect luck alone to produce, given how you searched. Bailey and López de Prado¹ give it a closed form: SR* = √V × z*(N), where N (trials) is the number of strategy results in your search and V (trial variance) is the variance of their Sharpe ratios. In plain terms: N is how hard you searched, V is how spread out the crowd was, and SR* is the score the luckiest member of that crowd would post even if nothing worked.

The Deflated Sharpe Ratio (DSR) then grades each individual strategy against that bar instead of against zero. A DSR near 1 means the result is hard to explain as the luckiest draw from your own search; a DSR near 0 means luck explains it fine. Sitting exactly at the bar means a DSR of 0.5, so "above the bar" and "DSR above 0.5" are the same statement.

The crucial asymmetry sits inside that formula. The count term z*(N) grows logarithmically: it barely reacts to more searching. The variance term √V scales the bar one-for-one, and it reacts to composition: what you put in the pool. We covered the count term at scale in our 100 million backtest study; this experiment isolates the other dial. (The two scans are consistent with each other: at their own scales, neither scan's best single strategy with 30 or more trades cleared 0.95.)

One unit note before the numbers: all Sharpe figures in this article are per-trade, computed on per-trade returns, because that is the unit SR* and the export's srObs column share. The scanner UI displays a much larger annualized figure (per-trade Sharpe scaled by √252, a convention that treats each trade as one observation), so do not compare on-screen Sharpes to the per-trade numbers here; the bar-versus-strategy comparison works the same in either unit. For scale, the strongest strategies in this article sit at 0.6 to 0.7 per-trade; whether a score like that means anything is exactly what the bar decides.

3. The experiment: one search, run twice

Market	BTC/USDT and ETH/USDT perpetuals (Binance USDT-M)
Window	49,999 hourly bars per symbol: 2020-10-26 00:00 to 2026-07-10 07:00 UTC (~5.7 yr)
Timeframes	indicator legs evaluated on 1h and 4h
Entries	every pair of two different indicator variations, both legs' conditions required together2
Exits	36 per pair: TP {3,4,5,6}% × SL {2,3,4}% × TTL {120,240,480} 1h bars, i.e. 5/10/20 days
Costs	taker fee 0.04%, slippage 0.01% (funding on perpetuals is not modeled)
Validation	walk-forward, a single ~90/10 split
Export	full export ("Save all results" in the scanner) for both runs

Run A used 10 indicators × 2 timeframes = 20 slots, 1,176 indicator variations in total: 690,900 entry pairs × 36 exits = 24,872,400 results per direction. The exact sweep, indicator by indicator (entry conditions are the scanner's automatic defaults per indicator; the same sweep runs on both timeframes except where noted):

Indicator	Parameter sweep	Variations per timeframe
Bollinger Bands	period {20→50, step 5} × std dev {1→4, step 0.5}	49
Connors RSI	RSI period {2,3,4,5} × streak period {2,3,4,5} × rank period {100,120,160,200}	64
Ichimoku	tenkan {7,9,11,12} × kijun {22,26,30,33} × senkou B {44,52,58,65}	64
Klinger	fast {30,34,38,42} × slow {50,55,60,65} × signal {7,13,20,27}	64
MACD	fast {5,12,19,26} × slow {13,26,39,46} × signal {5,7,9,11}	64
Parabolic SAR	AF start {0.01→0.04} × AF step {0.01→0.04} × AF max {0.1→0.4}	64
SMA	period {20→300, step 10}	29
Squeeze Momentum	BB period {15,20,30} × BB mult {2,3,4} × KC period {15,20,30} × KC mult {1.5,2}	54 on 1h / 36 on 4h (BB mult {2,3})
Stochastic RSI	RSI period {7,14,21} × stoch period {7,14,21} × K smooth {3,4,5} × D smooth {3,4,5}	81
Ultimate Oscillator	short {5,7,10,14} × mid {10,14,21,28} × long {20,28,40,56}	64

That is 597 variations on 1h plus 579 on 4h, the 1,176 in run A.

Run B removed every swept Ultimate Oscillator and Connors RSI configuration from both timeframes (4 of the 20 slots) and changed nothing else in the grid³: 920 variations, 422,740 pairs, 15,218,640 results per direction.

The removed configurations are the manipulated variable. In the parameter ranges this grid swept, they fire rarely on this market and these timeframes, so they flood the pool with trials that barely trade. One accounting note: not every row yields a valid trial, because a result with almost no trades cannot produce the statistics, so the trial counts below (20.7M and 21.9M for run A long and short, which fire differently by direction, and 14.8M for run B in both) are smaller than the row counts. Everything below is computed from those two exports: the per-row columns and the population metadata (trials, variance, SR*) that ship inside the file. No sampling, no estimates.

4. We called the shot first

An experiment on your own product invites a fair suspicion: did you run it until it said what you wanted? So we inverted the order. Before run B executed, we computed what its numbers should be, using only run A's export file (no rescans, no external data), and wrote them down. Run B ran minutes later.

	Predicted from run A's file	Run B measured
Rows per direction	15,218,640	15,218,640
Trials N (long / short)	14,806,909 / 14,796,184	14,806,909 / 14,796,184
Luck bar SR* (long / short)	0.397424 / 0.378341	0.397424 / 0.378341
Rows above the bar, among trades ≥ 30	3,798 / 840	3,798 / 840
DSR of the strategy with the best raw per-trade Sharpe (trades ≥ 30)	0.9125 / 0.9188	0.9125 / 0.9188

(That last row is the search's best by raw Sharpe. The champions in Section 5 are the best by deflated Sharpe: nearby scores, different rows.)

Every walk-forward-independent number we called in advance landed: row counts and trial counts to the digit, the trial variance to eight decimal places and the bar to seven, the above-bar counts exactly, the best raw-Sharpe strategy's deflated Sharpe to four digits.³ And across the 29.6 million trials the two runs share, per-trade Sharpe and trade count are bit-identical in every row: the engine is deterministic, so nothing but the crowd changed.

The formula is just as checkable in reverse. Applying the published z*(N) expression to each file's own trial count and variance reproduces the SR* recorded in that file to ten significant digits, on all four exports.

The export is a receipt. What the prediction proves is narrow: not that the metric is wise, but that the luck bar is pure arithmetic on the file, deterministic and recomputable by anyone who holds it. That is the property that matters in practice: if you hold one scan's receipt, the next scan's bar is computable before you press run.

5. Same trades, two verdicts

Here is what the composition dial did to individual strategies. These rows are bit-identical between the two runs: same entries, same exits, same fills, same per-trade Sharpe.

	srObs (per-trade)	Trades	PSR	DSR in run A	DSR in run B
Best long, by deflated Sharpe (two Bollinger Bands legs)	0.5954	90	0.99999	0.029	0.925
Best short, by deflated Sharpe (two Bollinger Bands legs)	0.7206	48	0.9984	0.035	0.919

Run A's population: 20.7M trials (long), 21.9M (short). Run B's: 14.8M for both.⁴

The Probabilistic Sharpe Ratio (PSR), which grades a track record against zero, is identical in both runs, because the track record is identical. The Deflated Sharpe flips from "luck explains this fine" to "luck mostly does not," because the crowd the strategy stood in changed. PSR doesn't know what you searched; DSR does. Neither metric is wrong: they answer different questions, and only one of those questions involves your search.

The strategy never changed; the crowd it stood in did.

The luck bar SR* for the same search run twice: run A with all 20 indicator-timeframe slots, run B with the four rare-signal slots removed. The dashed marker is run B's best raw per-trade Sharpe among strategies with 30 or more trades; the identical trades clear run B's bar and sit far under run A's. Source: two-run A/B search, BTC and ETH perpetuals, 2026-07-10.

6. The weak dial and the strong dial

SR* factors cleanly, so we can say exactly which dial moved it.

Component	Long	Short
Trial variance term √V	fell 2.13x (V itself: 0.0248 → 0.0055)	fell 3.04x (V: 0.0457 → 0.0050)
Count term z*(N)	fell 1.011x (5.436 → 5.376)	fell 1.013x (5.447 → 5.376)
Bar SR* in total	fell 2.15x	fell 3.08x

The products check: 2.13 × 1.011 = 2.15, and 3.04 × 1.013 = 3.08.

Run B dropped 28 percent of the long trials and 33 percent of the short trials, and the count term barely noticed: about 1 percent, the logarithmic indifference from Section 2. The variance term is where the entire move lives. Trials that rarely fire produce Sharpe estimates computed on 3, 5, 8 trades, and those estimates are wild: they scatter across a huge range and stretch the population's variance. Remove them and the crowd tightens.

The bar didn't drop because we searched less; it dropped because the lottery tickets left the pool.

Every valid trial with 2 to 150 trades, both directions, as a density field: shade shows how many strategies land in each (trade count × DSR) cell. Under run A's crowd the best deflated Sharpe among 30-plus-trade strategies is 0.31; under run B's, the same style of cloud lifts and the best reaches 0.925, still under the 0.95 line. Source: run A and run B exports, 2026-07-10.

7. The firing-rate audit, and why it's not the indicator

Which trials were the lottery tickets? Define a ghost trial as a strategy result with fewer than 10 trades, among the valid trials that entered the population statistics. Then audit run A by indicator-timeframe slot:

Slot (run A)	Ghost share	Median trades
Ultimate Oscillator @ 4h	21.8%	24
Ultimate Oscillator @ 1h	16.9%	38
Connors RSI @ 4h	16.5%	35
Connors RSI @ 1h	9.9%	86
The other 16 slots	1.3% to 4.6%	405 to 1,651

The four removed slots are not marginally worse. Their median trade counts sit at 24 to 86 against 405 or more everywhere else, in these configurations, on this market and window.

But the audit gets more useful one level deeper. Within a single indicator, ghost share is not a property of the indicator; it is a property of the parameters. Among run A's long-side Connors RSI variations on 1h, ghost share runs from 1.5% to 29.7% depending on parameters: the (2,2,120) configuration fires almost like a mainstream indicator, while stricter cousins barely trade. The Ultimate Oscillator on 4h, again on the long side, spans 20.9% to 100%, and 30 of its 64 swept variations produced no valid trials at all. Same indicator, same market: the firing range is set by the configuration, not the name on the card. We saw the same thing when combining strategies: the configuration decides what a component contributes.

The fix is not deleting an indicator from your toolbox; it is tuning a configuration until it actually fires.

Top: ghost-trial share per indicator-timeframe slot in run A (both directions, valid trials). Bottom: the same share for every swept parameter variation inside the four removed slots, long side. The spread inside one indicator runs from healthy to never firing, so the audit condemns configurations, not indicator names. Source: run A export, 2026-07-10.

8. What this does not mean

• It is not an indicator blacklist. The spectrum above is the evidence: Connors RSI at (2,2,120) on 1h fired fine in this configuration. The audit condemns configurations that rarely fire in your grid, not indicator names.
• Run B did not "find edges," but it did hand over the raw material for one. Its best single results among strategies with 30 or more trades, DSR 0.925 (long) and 0.919 (short) against 14.8M trials, sit below 0.95, the conventional reference line, and this is one window on one market pair. The 372 long and 137 short strategies that passed a four-condition screen (30-plus trades, above the bar, positive walk-forward, no negative fold) are candidates that earned a second look, not proven edges. And the way past the line is the one from our pairing study: pooling the best long with the best short. Here is the shortlist:

	Entry (two legs)	Exit	Trades	Win rate	Return (5.7 yr)	Max DD	DSR
Best long	Bollinger(25, 3.5) @1h + Bollinger(40, 2.5) @4h	TP 3% / SL 4% / TTL 120 bars	90	72.2%	+187%	−9.2%	0.925
Best short	Bollinger(45, 3.5) @1h + Bollinger(40, 4) @4h	TP 6% / SL 4% / TTL 480 bars	48	83.3%	+53%	−4.7%	0.919
Pooled long + short	both of the above, one stream	as above	138	76.1%	~+340%	not re-run	~0.97

The pooled row is an arithmetic estimate from the export's own moments, compounded return, trade-weighted win rate, and composite deflation against the same 14.8M-trial bar, not a re-backtest. It clears the conventional line with zero additional backtests: combining evidence, again, beats searching harder. Costs are modeled as in Section 3, and note what that excludes: a live perpetual position also pays or receives funding, which this scan does not model, and a TTL-480 hold can run to 20 days. This is one window on one market pair, and a DSR is an evidence level, not a promise of profit.

And run the scale check every crypto trader asks for: neither champion's raw return beats buy-and-hold. Just holding BTC over a near-identical window returned about +443%, with a −77.6% drawdown along the way (the 100M study runs this check in full). What these rows offer is not outperformance; it is a −9.2% and a −4.7% drawdown, and an evidence trail.

• Isn't run B itself hindsight? A fair challenge: we removed those four slots after seeing run A. Three things separate this from laundering a result. First, the removal rule keys on trade counts, never on returns: firing rate is an input property, auditable from a cheap dry-run before any full scan exists (Section 9), and it does not know whether a trial made money. Second, no search was re-graded from the inside. Run A's zero-above-bar verdict stands, deflated against its own full 20.7M and 21.9M-trial populations, and run B is graded against its own complete 14.8M, nothing deleted from either. The illegitimate move is pruning weak trials out of a finished search and recomputing that same search's bar, and it leaves tracks: the trial count and variance in the file would no longer match the claim. Third, the outcome was never in doubt, which is why run B's numbers could be computed before it ran; this experiment demonstrates a mechanism, it does not shop for a result. Even so, run B's champions inherit some hindsight on this window, one more reason we do not call them edges: the rule earns its keep on the next scan, the one designed before it runs.
• Thresholds are conventions, and here is the full picture. Counting both directions of run B by the export's native dsr column (which deflates against the per-row trial count, the file's numTrials):

Trades	DSR > 0.5	> 0.8	> 0.9	> 0.95	> 0.99
30 to 99	4,438	246	10	0	0
100+	200	4	0	0	0

One robustness note: counting trials per entry pair instead of per row (N ≈ 411,000) raises the bar to 0.539 (long) and 0.467 (short), and the best singles under that stricter counting, among strategies with 30 or more trades, land at 0.780 and 0.850. Still below every conventional line: the conclusion does not depend on how you count N.

9. Design the firing range, then read the receipt

What this buys you in practice is a pre-scan checklist:

1. Set the firing range first. Before scanning, ask of each indicator configuration: does it actually produce signals at this timeframe, on this market? That is a property you control in the grid.
2. Dry-run small and read the trade-count distribution. A cheap scan shows you which slots are ghosts before you spend a big one.
3. After the scan, read the population before the leaderboard. Trials N, trial variance, SR*: those three numbers say what kind of search your results came from.
4. Then read each row's deflated Sharpe against that bar, with its trade count next to it.

None of this is scale-bound. Fifty variations tried by hand have a luck bar too; the difference is that nothing wrote it down. The receipt is what writing it down looks like.

The Backtest Search Receipt

Every number this article used ships in a Rulyfi full export on any paid plan: per-row columns plus population metadata in the file itself (free exports carry the basic result columns only). Read together, they answer the questions that decide whether a backtest means anything:

Field	The question it answers
tradeCount (per row)	Did this strategy trade enough for its stats to mean anything?
Trials N (file metadata, numTrials)	How hard was the search this row came from?
Trial variance V (file metadata, varSr)	How spread out was the crowd, and was it inflated by ghosts?
SR* (file metadata, srStar)	What score would pure luck have posted in this search?
dsr (per row)	Does this row beat that luck, specifically?

That receipt is what made Section 4 possible, and it is machine-readable: the same audit runs programmatically over the API if your workflow is code instead of a download folder.

So, how many backtests is too many? That was never the number worth losing sleep over: dropping about a third of the trials moved the count term of the bar by about one percent. What decides your hurdle is the crowd you assemble, and both dials are printed on the receipt your scan already hands you. A wide search was never the sin; a blind one is, and your export already carries every number you need to run yours with eyes open.

Frequently asked questions

Can I delete the low-trade trials after the scan to lower my hurdle? No. Removing trials after seeing results is the selection bias DSR is built to correct, and it leaves tracks: the trial count and variance are recorded in the export, so a recomputed bar will not match your claimed one. Design the grid before scanning instead; that is the legitimate version of the same move.

Why did PSR stay identical while DSR flipped? Because they answer different questions. PSR grades a track record against zero and only looks at that strategy's own returns, which were bit-identical in both runs. DSR grades the same track record against the best result luck would produce across everything you searched. Both are valid; only DSR knows the search existed.

Should I stop using the Ultimate Oscillator or Connors RSI? No. In this grid's swept ranges they fired rarely on this market and these timeframes, but the spread within each indicator was enormous: from configurations that never traded to ones with healthy trade counts. The actionable lesson is to check a configuration's firing rate on your market and timeframe before giving it seats in a large search.

---

Auto-trading and trading carry a risk of losing your principal. This article is educational, does not guarantee profit, and past backtest results do not predict future returns.

Footnotes

1. Bailey, D. H., & López de Prado, M. (2014). "The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality". The Journal of Portfolio Management, 40(5). ↩

2. Entry conditions are the scanner's automatic defaults per indicator, and a trade requires both legs' conditions at once. Execution mechanics, fill conventions, and their caveats are those of the same engine documented in our 100 million backtest study. ↩

3. Walk-forward-dependent numbers are excluded from the exact-match list for a mundane reason: the two runs' in-sample/out-of-sample boundaries differ by about 55 bars (the UI slider stores an unrounded fraction, 0.9006 vs 0.9017), so fold-based figures are quoted against each run's own split. The four-condition screen row counts, predicted 1,997 (long) from run A's folds, measured 2,005 under run B's folds, land on identical strategy counts: 372 long, 137 short. ↩ ↩²

4. PSR values here are computed directly from the export columns. Sharpe-type figures are capped at the extremes (Sharpe at ±100, per-trade Sharpe at ±6.2994), and a PSR that reads as exactly 1.0 is rounding, not certainty. ↩