Scanner

Your AI's Most Spectacular Backtest Is Probably a Mirage: How a Scan Tells a Real Edge From a Lucky One

Q: Why can't a single backtest catch the mirage?

Because the mirage is only visible against the search that produced it. A backtest grades one strategy against zero and returns the Probabilistic Sharpe Ratio, which for our mirage was a confident 0.975. The Deflated Sharpe grades that same strategy against the luckiest result you would expect from all the variants you tried, and that needs the whole population. It is a scanner metric for a mathematical reason, not a marketing one.

Q: Does a high Deflated Sharpe mean the strategy makes money?

No, and treating it that way is the mistake the whole method is built to prevent. A high Deflated Sharpe means a result is unlikely to be luck; it is an evidence level, not a signal. Even the rare survivors above were measured in-sample, carry their own drawdowns, and come with the same disclaimer as everything else: past results, deflated or not, do not predict future returns. Treat a high dsr as a candidate for paper trading and small live size, not a green light.

Q: Can an AI agent run this whole loop on its own?

Yes, that is the point of the MCP server. An agent can generate a strategy, validate it, scan its parameter neighborhood, read the deflated leaderboard, pull the full population, run the gauntlet query, and even pool complementary survivors, all through tool calls, with Claude Code, Claude Desktop, Cursor, or any MCP client. You review the shortlist and its caveats instead of clicking through a builder.

2026.07.08·12 min read·Rulyfi

Key Takeaways

Generating strategies is nearly free now, and the proudest-looking result is usually the most dangerous. A single backtest can only give you the Probabilistic Sharpe Ratio (PSR), which grades against zero; a scan computes the Deflated Sharpe Ratio (DSR), which grades the winner against the whole search that produced it.

Live, one run_scan call deflated a Sharpe 4.514 short (PSR 0.975) to a Deflated Sharpe of 0.000, because it was the luckiest of 893 variants. At the extreme, in our study of nearly 100 million backtests, the single most spectacular curve, up 684% at Sharpe 5.59, deflates to 0.10. The flashiest number is the least trustworthy one.

Real edges exist, but they are rare and modest. In that same study, 203 single strategies scored a Deflated Sharpe between 0.8 and 0.9, but not one cleared 0.9, and only pooling a long with a short reached 0.9502. A real edge is a line you cross by a hair, not a cliff you clear.

This is a workflow, not a screenshot. An agent runs the whole funnel over MCP: run_scan a neighborhood, download_export the population with its per-row dsr column, filter, and even pool complementary survivors, the same gauntlet the 100-million-backtest study ran by hand.

Your AI coding agent can write and backtest a thousand trading strategies before lunch, and its proudest result is usually its most dangerous. Watch it happen live: the agent hands you a Bitcoin short with a Sharpe of 4.514 and a Probabilistic Sharpe Ratio of 0.975, a textbook near-certainty. One run_scan call deflates it to a Deflated Sharpe of 0.000. The flashiest backtest is the fake, and it is not a quirk of one strategy: in our study of nearly 100 million backtests, the single most spectacular equity curve, up 684% at a Sharpe of 5.59, also sits below the luck line. This article shows the scan that catches the mirage in real time, what a genuine edge actually looks like (rare, modest, and real), and the agent loop that runs the whole funnel for you.

The two contenders

An agent that tests thousands of strategies will always hand you its best-looking one, and "best-looking out of thousands" is precisely the recipe for a fluke, because the resume never shows the search behind it. Contender one is that flashy MACD short, the mirage. Contender two is the harder, more useful thing: a genuine edge, once you know what one actually looks like. The only measurement that separates them is the one a single backtest cannot compute, because it takes the whole population to calculate: the Deflated Sharpe. Start with the mirage, because it shows exactly what that number does.

Case A: the flashiest backtest is the fake

The Probabilistic Sharpe Ratio asks whether a strategy's true Sharpe is above zero, given its sample size and the shape of its returns. It is the honest ceiling of a single backtest: the best a lone strategy can say about itself. A PSR of 0.975 reads as "97.5% confident this beats doing nothing," and that confidence is exactly the trap.

The MACD short did not arrive alone. The agent cast a wide net across momentum indicators and kept the best result, something like:

run_scan({
  cards: [
    { type: "MACD", params: { fast: { from: 8, to: 16, step: 2 }, slow: { from: 20, to: 30, step: 2 }, signal_period: [9] } },
    { type: "RSI", params: { period: { from: 7, to: 21, step: 2 } } },
    { type: "STOCHASTIC_RSI", params: { rsi_period: [14], stoch_period: { from: 10, to: 20, step: 5 } } }
  ],
  combo_size: 2,
  exit: { tps: [3, 4, 5], sls: [2, 3], ttls: [48, 96] },
  symbols: ["BTCUSDT"],
  exchange: "binance",
  timeframe: "1h",
  direction: "short",
  min_trade_count: 30
})

That sweep evaluated 893 variants, and the MACD short was the single best of them. Here is the move a plain backtest cannot make: because the scan holds all 893 results, it can compute how good the best one would look by luck alone after 893 tries, and grade the winner against that bar instead of against zero. That bar is the Deflated Sharpe Ratio.¹ For this winner it comes out to 0.000: essentially no chance its edge is real rather than the luckiest draw of 893.

A 4.5 Sharpe scoring 0.000 looks like a contradiction until you see which Sharpe the deflation weighs. The 4.514 is an annualized headline, the number a leaderboard sorts on. The Deflated Sharpe works on the per-trade evidence instead: it grades the winner's per-trade Sharpe against the luckiest per-trade result you would expect from 893 tries, then adjusts for how few trades and how lopsided a track record it has. A loud annualized figure routinely hides a per-trade edge that sits below that luck line, and drawn as the single best of 893, this one sat far below it. Counted honestly, the chance it is skill rather than the best fluke of the search rounds to zero, so do not read the annualized 4.5 as if it clears the bar directly. get_scan_results returns the leaderboard with that DSR column filled in, so the agent sees the 0.000 next to the 4.514 in the same response. The mirage is caught in one call, before a cent is risked.

Now scale it up, because the pattern only gets starker. Our study of nearly 100 million Bitcoin backtests ran the same deflation across tens of millions of strategies per direction. There, the single most spectacular result, a long at +684%, a Sharpe of 5.59, and a PSR of 0.99986 (about as certain as that test ever reads), has a Deflated Sharpe of about 0.10. The loudest equity curve in a hundred million, drawn from that many tries, still sits below the luck baseline. As that study puts it, return is the loudest metric and the least trustworthy. That is the whole reason to stop trusting a single number.

Why this is the whole game

Strategy generation used to be the hard part. It is now the cheap part. An agent can produce ten thousand plausible strategies overnight, and a chunk of them will backtest beautifully for the same reason a chunk of lottery tickets win: you bought a lot of tickets. The bottleneck has moved. The question is no longer "can I find a strategy with a great backtest," it is "of the thousands my agent just found, which are real and which are the luckiest tickets."

That is exactly the question a single run_backtest cannot answer and a run_scan can. A backtest grades one strategy against nothing. A scan grades the winner against the entire search that produced it. The scanner's job is not to find you a winner. It is to tell you, cheaply and before live trading does it the expensive way, whether the winner in front of you is skill or luck.

Case B: what a real edge actually looks like

If the flashy ones are fake, does anything survive? Yes, but it is rarer and plainer than a leaderboard would ever suggest. The honest way to see it is in the full population of that 100-million-backtest study, sorted not by return but by deflated confidence. Here is where the survivors actually sit:

Deflated Sharpe band	Single strategies
0.5 to 0.6	1,577
0.6 to 0.7	917
0.7 to 0.8	433
0.8 to 0.9	203
0.9 and up	0

Read that bottom row twice. Out of nearly 100 million backtests, 203 single strategies earned a Deflated Sharpe between 0.8 and 0.9, real, genuinely unlikely to be luck, and not one cleared 0.9. The strongest individual strategies stall at 0.871 on the long side and 0.898 on the short, each a roughly 4.3-sigma result that is about a one-in-a-hundred-thousand find in isolation, and still short of the 0.9 wall. Real edges are not a myth. They are just this rare, and this modest.

So how does anything reach the strict 0.95 bar? Not by searching harder for a better single strategy, but by combining evidence. In that study, the only thing to cross 0.95 was a pooled pair, a long and a nearly independent short, scoring 0.9502: a long MACD(16,13,5) + BB(50,2) with 37 trades combined with a short MACD(16,37,6) + StochRSI with 35 trades, 72 trades in all. As the study says, 0.95 is the mirror of p < 0.05, a convention rather than a cliff, and the shape of a real edge is not something you clear with room to spare but a line you cross by a hair. That is the honest picture the scanner paints: a spectrum of confidence, a handful of rare survivors, and one combined idea grazing the top.

Scan years, not months

There is a wrong way to run any of these scans, and it is the tempting one, so it is worth calling out. Aim a scanner at just the last several months, a stretch where Bitcoin mostly fell, and a short strategy wins easily. It looks like an edge. It is not: it is one strategy riding one downtrend, and because a short walk-forward window keeps every fold inside that same falling market, the out-of-sample check has nothing to catch it with. A strategy that only works in one regime has captured that regime, not found an edge. This is the most common way a careful-looking backtest still lies: the window was too short to contain a market that changed its mind. It is also why the study above ran across more than five years, spanning a bull market, a bear market, and a recovery, so a survivor had to hold up in more than one kind of world. When you point the scanner at your own idea, give it years, not months.

The other trap: three trades and a fake 58 Sharpe

A population statistic is one of two hygiene levers. The other is a floor on evidence. Point a scan at an RSI-and-moving-average neighborhood, sort the 4,174 results by raw Sharpe with no floor, and the "winner" is absurd: RSI(11) + SMA(100) at a Sharpe of 58.65, a payoff ratio pinned at its display cap of 999, and a Deflated Sharpe of 0.96 that even looks respectable.

It made three trades. At three trades the skew and kurtosis terms that do the deflating have nothing to work with, so the number is noise wearing a suit. This is why every run_scan above carried min_trade_count: 30. The floor removes results too small to judge before any ranking happens. A real scan needs both levers at once: a population luck bar so a cherry-picked winner is exposed, and a trade floor so a tiny-sample fluke never becomes the winner in the first place.

The loop an agent runs while you sleep

Here is the part that makes this a workflow rather than a lecture. The 100-million-backtest study ran its funnel by hand, once. An agent runs the same funnel on your own ideas, on a schedule, using four of the seven MCP tools:

Generate and validate. The agent turns an idea, from Python, TradingView, or plain English, into the strategy format, then calls validate_strategy to fix it field by field.
Scan, not backtest. run_scan sweeps the neighborhood over a long, multi-regime window. Every result carries its Deflated Sharpe and the trial count it was judged against.
Pull the whole population. get_scan_results reads the top-K leaderboard, and download_export streams the full parquet, every variant with its per-row dsr column, to local disk.

download_export({ scan_id: "scan_..." })
// -> ./rulyfi-exports/scan_.../merged.parquet  (every variant + its dsr column)

With the population local, the agent runs the exact gauntlet from the study, as one query:

SELECT count(*) FROM 'rulyfi-exports/scan_.../merged.parquet'
WHERE tradeCount >= 30 AND dsr > 0.5 AND wfEfficiency > 0 AND forwardNegFolds = 0;

And it can take the last step the study did too: the raw ingredients ride along in the same file, so an agent can combine a long survivor with a nearly independent short in closed form, no extra backtests, and check whether the pooled pair reaches the 0.95 that no single strategy did. Generate wide, deflate hard, throw almost everything away, and pool the rare survivors. One scope note: run_backtest is spot-market only, while scans cover spot and futures, walk-forward across the grid, and this full-population export. Single backtests are the starting point; the scan is the product, and it all runs from an agent at api.rulyfi.com/v1 while you are not watching.

What the scanner actually gives you

It does not tell you your strategy is wrong. It tells you which of your strategies are worth a second look, and it shows you exactly how rare that is. That is a capability, not a scolding, and in an age where an agent can generate strategies faster than any human can vet them, it is the missing half of the workflow. Anyone can produce a Sharpe 5 backtest now. The scarce thing is the one statistic that says whether that 5 survived the search it came from, and it ships as a per-row dsr column in every full-export on a paid plan, so the whole funnel above runs on your own scans, not just ours.

Real edges are rare, and it is worth being clear-eyed about that: in that hundred-million-backtest run, only 3,130 strategies, about 0.003%, cleared a full gauntlet of deflation and out-of-sample checks, and only a pooled pair reached the strict bar. Rare is not zero, and a tool that tells you where your idea sits on that spectrum, before you fund a mirage, is worth more than one that promises winners. Parameter sweeping itself is not new; tools like vectorbt and QuantConnect will run a grid for you. But a grid search points the wrong way, hunting for the single best parameter, which is an overfitting machine by design. The scanner inverts it: it uses the whole population as the antidote, computing the deflated bar that tells a real edge from a lucky one, and it hands that to a no-code builder and an AI agent alike.

The developer reference, endpoints, authentication, and rate limits, lives at /developers; the strategy format, with its 103-indicator catalog and condition grammar, is documented in full at strategy-schema.md. The full method behind every number here is in our study of nearly 100 million Bitcoin backtests.

Frequently asked questions

Why can't a single backtest catch the mirage? Because the mirage is only visible against the search that produced it. A backtest grades one strategy against zero and returns the Probabilistic Sharpe Ratio, which for our mirage was a confident 0.975. The Deflated Sharpe grades that same strategy against the luckiest result you would expect from all the variants you tried, and that needs the whole population. It is a scanner metric for a mathematical reason, not a marketing one.

Does a high Deflated Sharpe mean the strategy makes money? No, and treating it that way is the mistake the whole method is built to prevent. A high Deflated Sharpe means a result is unlikely to be luck; it is an evidence level, not a signal. Even the rare survivors above were measured in-sample, carry their own drawdowns, and come with the same disclaimer as everything else: past results, deflated or not, do not predict future returns. Treat a high dsr as a candidate for paper trading and small live size, not a green light.

Can an AI agent run this whole loop on its own? Yes, that is the point of the MCP server. An agent can generate a strategy, validate it, scan its parameter neighborhood, read the deflated leaderboard, pull the full population, run the gauntlet query, and even pool complementary survivors, all through tool calls, with Claude Code, Claude Desktop, Cursor, or any MCP client. You review the shortlist and its caveats instead of clicking through a builder.

Auto-trading and trading carry a risk of losing your principal. This article is educational, does not guarantee profit, and past backtest results do not predict future returns.

Bailey, D. H., & López de Prado, M. (2014). "The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality". The Journal of Portfolio Management, 40(5). ↩

scannerdeflated-sharpeai-agentmcpwalk-forward