Statistics

How Many Backtests Is Too Many — Why the Answer Depends on How Many Times You Looked

You changed an SMA pair, ran the test again, and kept the better result. Do this a few dozen times and the question stops being whether the rules are good and becomes whether the search was honest. There is no universal answer to "how many backtests is too many" — not because the question is vague, but because the limit moves with the length of your sample, the independence of the variants, and the Sharpe you are claiming.

This article shows how the limit is actually computed: why raw run counts overstate the problem, why the luckiest variant of a large search looks impressive even with no edge, and how a trial counter turns that into a threshold you can read instead of a lecture about discipline.

There is no universal limit

The most-cited starting point is Bailey and Lopez de Prado: on five years of daily data, trying more than roughly 45 independent variants already makes a Sharpe of 1 or higher likely to appear by chance alone. That number is not a law of backtesting. It is one point on a surface with three variables.

Sample length. Shorter samples have noisier Sharpe estimates, so each additional trial adds more potential luck. A year of daily data tolerates far fewer tries than a decade of the same data before producing the same false confidence.

Independence. Forty-five truly independent ideas are not the same as forty-five parameter settings of one idea. Trials that share structure collapse toward each other, and the effective count is what matters.

The claimed Sharpe. A modest claim survives more searching than a spectacular one. The bar is not fixed at "N tests"; it is the interaction of all three.

So the honest answer to the title question is a formula, not a number. What remains is estimating the two inputs you can actually observe: how many trials you ran, and how much of their similarity counts as one idea.

Raw runs versus effective trials

A 0.40% stop and a 0.41% stop are almost the same rule. Running both and keeping the better one is not two independent experiments; it is one experiment with a rounding difference, and counting it as two inflates your penalty while teaching you nothing. The same applies to a fast moving average of 10 versus 11, or a take-profit of 1.5% versus 1.6%.

The standard fix is a participation ratio computed from the eigenvalues of the returns matrix: runs whose equity curves move together count as a fraction of one trial, not as one each. A set of 18 runs whose curves are heavily overlapping collapses to 11 effective trials — an interface example of the kind the trial counter is designed to show, not a result from any live account.

This cuts both ways, and it is worth being clear about that. A low effective count is the milder verdict: it means your search was small and correlated, so little correction is owed. A high effective count next to a modest Sharpe is the verdict that hurts, because it says you searched broadly and the thing you found is not distinctive. Either way, the number exists whether or not anyone writes it down — the counter only stops you from pretending otherwise.

The expected best of luck

Why does searching produce good-looking results with no underlying edge? Because the maximum of many draws rises even when the draws are centered on zero. Roughly, the expected best of N noise draws sits about √(2 · ln N) standard deviations above the mean, and the standard error of the Sharpe plays the role of that standard deviation.

Our own landing slider uses the interface approximation 0.45 · sqrt(2 · ln N), where 0.45 stands for the noise scale of a short, trade-level sample. It is a design approximation, not a statistical guarantee, and it exists to make the shape of the problem visible:

Tried variants Nsqrt(2 · ln N)Expected best (slider approximation)
10.000.00
102.150.97
1003.041.37
10003.721.67

Read the last row as a warning about selection, not as a forecast: after 1000 parameter combinations, the single luckiest one shows a Sharpe near 1.67 purely because it is the maximum of 1000 draws. Nothing about the rules improved. The Sharpe ratio itself is honest — it just reports the value of the variant you happened to keep.

A worked number for your own sample

Take a five-year daily sample, T ≈ 1260, with a claimed Sharpe of 1.15. The standard error of that estimate is roughly sigma_SR = sqrt((1 + SR^2 / 2) / T) = sqrt(1.66 / 1260) ≈ 0.036. You tried 60 parameter sets that collapsed into 40 effective trials. The luck bar is 0.036 × sqrt(2 · ln 40) = 0.036 × 2.72 ≈ 0.10 — so roughly a tenth of the reported 1.15 is what picking the best of 40 produces by itself.

Now rerun the identical search on one year of daily data: T = 252, sigma_SR rises to about 0.081, and the same 40 effective trials give a luck bar of 0.081 × 2.72 ≈ 0.22. Same rules, same count, twice the noise. That is why "I only tried forty things" is not a defense on a short sample.

Effective trials N_effsqrt(2 · ln N_eff)Luck bar, 5 years of daily dataLuck bar, 1 year of daily data
51.790.060.15
112.190.080.18
402.720.100.22
1003.040.110.25
4003.460.130.28
10003.720.130.30

The columns shrink relative to each other as the sample grows, which is the whole argument for longer tests: they do not remove luck, they make it cheaper.

The deflated bar in one screen

Subtracting the expected best of luck from the observed Sharpe gives the deflated Sharpe, and the practical output is a bar: the value your run must clear at the given trial count before the result counts as evidence. The engine behind it runs on a strict requirement — a deflated Sharpe of at least 0.95 — and the interface states the plain-language version.

As an interface example, the figures the UI is designed to display rather than results from a live account: a run reporting Sharpe 0.94 against a required 1.47 at 11 effective trials, or a gate phrased as "need 1.90 at 41 trials." Both say the same thing in different clothes — the metric is judged against a bar that moved because you searched, not against zero.

Two things this does not do: it does not forgive the run, and it does not reset when you close the tab. A high deflated requirement is information about the search, not a punishment attached to the strategy. When the bar is unreachable, the honest moves are a longer sample or rules that are fewer and more distinct — both of which reduce N_eff from the other side. What determines whether the surviving result holds up is still the part of the data that never saw the search: independent testing.

Probability of backtest overfitting, in one paragraph

The deflated Sharpe asks whether the number is large enough. A second check asks whether the winner is stable: in combinatorially symmetric cross-validation, you split the data into blocks, pick the best rule on one half of the blocks, and check where that rule lands on the other half. If the champion of half the data falls below the median on the rest, you did not find a strategy — you found the luckiest block order. The share of splits where that happens is the probability of backtest overfitting, and the sane gate is at most 0.5: better than a coin flip at surviving your own resampling.

How Normal Trading counts

The platform treats the counter as bookkeeping, not as a suggestion:

  • Every Run increments the count. There is no "practice" category that stays off the books.
  • Similar variants collapse into effective trials, so padding the search with near-duplicates does not inflate the denominator either.
  • The counter cannot be zeroed, and a paid plan does not soften the verdict — the bar is computed from the search, not from the subscription.
  • The Research report stays silent about strategy quality and simply reports what was measured; the verdict belongs to the threshold, not to a score.

The counter is a protection rather than a paywall. A search you cannot count is a search that will always tell you what you wanted to hear, which is exactly how unaccounted frictions and optimistic fills survive review in the first place.

How to check this yourself

  1. Write down every variant you ran for one strategy, including the ones you abandoned early. That list is your raw N.
  2. Collapse it honestly: variants differing only in a rounded parameter are one trial, not two.
  3. Check the sample length behind the claim. A high trial count on a short sample is the worst combination in the table above.
  4. Read the Sharpe next to the required bar for that trial count, never next to zero.
  5. Reject the run if the same rules fail on the half of the data that did not select them.
  6. Confirm the effect survives costs and one round of live-style fills before believing any of it.

To watch the mechanism rather than read about it, open the Replay simulator, duplicate a saved graph, change one period, and read how the trial count and the required bar move together.