The usual "last 30% of history" holdout fails three checks before it even runs. You have already scrolled the whole series and formed an opinion about its ending. The split is a percentage, so topping up data silently moves the boundary. And a trade opened in training often closes inside testing, which feeds test prices back into training decisions.
Out-of-sample testing is not a slice you cut once tuning is done. It is a protocol: freeze the dates at registration, drop the bars at the joints, store the rule as a hash, open the holdout exactly once. The order matters, and this article follows it.
In-sample vs out-of-sample: who saw what
The distinction is not a naming convention. It answers one question: did any part of the decision touch this data before the decision was made?
| Slice | Who has seen it | What may change here | How often it is opened |
|---|---|---|---|
| Train (in-sample) | You, freely, many times | Rule, parameters, filters, position sizing | Unlimited |
| Validation | You, after candidates are frozen | Choice between a few frozen variants | During tuning |
| Holdout | Nobody | Nothing | Once, at the verdict |
| Forward | Nobody — the data did not exist yet | Nothing | As time passes |
Training is where the strategy is allowed to be wrong: change parameters there as often as you like, it only costs you time. The holdout is where the question gets asked. Once a number changed because of something you saw in the holdout, that window has become training data, and no later statistic repairs it.
This is why re-tuning after a failed holdout read is the most common protocol violation in retail testing. The trader adjusts one filter, reruns, and reports the second result as if the first look never happened. Every additional look is an additional trial, and the trial count moves the bar you have to clear on how many backtests is too many.
A percentage split is not a seal
Three mechanics break a percentage-based split, and none of them requires bad intent.
First, appending data re-slices everything. "Last 30%" is a formula, not a window: when a new year of bars arrives, the boundary moves and windows that were sealed shift position. A holdout that moves under your feet was never a holdout. Second, prior browsing contaminates regardless of code. If the last stretch was already on your screen when you chose the filter, the information moved one way, and the split cannot unsend it. Third, a rule that is still editable after the split makes the split decorative — the edit is the leak, and the date on the chart will not show it.
Registration replaces formulas with dates. At registration the four slices are stamped — train, validation, holdout, forward — the rule is stored as a hash, and the opening of the holdout becomes a single recorded event. History added later extends only the forward track; the sealed window keeps its original dates, so a run produced a year from now is still judged on the same bars.
What a registration record should contain is short and checkable: the four date boundaries, the hash of the rule, the engine version, the cost profile, and the trial number this run adds. If any of those can change without changing the record, the seal has a second door — and a protocol you cannot audit is a protocol you cannot trust after the result disappoints you.
Purge and embargo at the joints
Boundary leak is mechanical. A rule with a 20-bar holding horizon, opened on the last training bar, closes 20 bars into validation. Its result is computed from prices the training window was supposed to be blind to, and any parameter choice made afterwards has quietly absorbed it.
Purge removes every training observation whose life interval [t, t+h] crosses into the next slice, where h is the holding or labeling horizon. Embargo then removes a strip of bars immediately after the joint, sized as the larger of about 1% of the window and the horizon itself. Both joints — train to validation, validation to holdout — get the treatment. The horizon cuts both ways: a feature whose label is built from the next 20 bars leaks just as hard as a position held for 20 bars, so the label window counts as h too.
Worked example: holding horizon h = 20 bars, training window of T = 6,000 bars. Purge drops the last 20 training bars. Embargo drops max(0.01 × 6,000, 20) = 60 bars after the joint. The total cost is 80 bars, about 1.3% of the window, which is cheap for a verdict that is not computed from leaked answers.
Walk-forward analysis or a stability scan
Rolling in-sample to out-of-sample folds are out-of-sample only under a condition: parameters are chosen separately inside each fold's own training window, or fixed before any fold exists. Then each fold tests a choice it was not allowed to make.
Worked example: six years of daily bars, a 12-month training window, a 3-month test window rolled monthly. Twelve folds later you have 36 months of stitched test curve, and every fold's parameters came only from its own 12 months of training data.
Two windows are defensible here. A rolling window keeps a constant training length and forgets the distant past, which suits instruments where the regime changes; an expanding window trains from the start of history, which suits slow strategies whose edge needs decades. Both are legitimate as long as the choice is made before results are read, and stating which one you used is part of the report.
The ratio to watch is the stitched test Sharpe divided by the in-sample Sharpe (see what the Sharpe ratio actually measures). A ratio below roughly 0.5 says the procedure is fitting mostly noise — a statement about the process, not about any single parameter value, which means fixing one parameter does not cure it.
If the fold boundaries were drawn after looking at the whole series, or parameters were re-chosen after seeing the stitched curve, what you have is a stability scan. It is genuinely useful for describing how a rule behaves across history, and it should be labeled that way in the report instead of being passed off as unseen evidence.
Three reports, in this order
Research first: is there an effect at all? Costs off, trial counter silent, cheap and disposable. Run second: costs applied, the trial counter incremented, fragility printed next to the headline number — our interface shows a Sharpe of 0.94 against a required 1.47 at 11 effective trials, an interface example from demo copy rather than a live account. Verdict third: forward versus expectation, stamped and not editable afterwards.
Passing Research and failing Run is the most common honest outcome, and it is the cheap place to lose. The three numbers worth reading in a Run are the metric net of costs (backtest costs), drawdown on the net curve (max drawdown explained), and the threshold that your trial count demands. A verdict that skips any of the three is a verdict about a curve, not about a strategy.
Blind forward: data that did not exist yet
The only honest reset button is called time. Freeze the rule, register it, then let months accrue without touching anything. The forward slice cannot be tuned into shape, because it was not there when the tuning happened; no discipline manufactures that property, only the calendar does.
A stamp on forward data answers a different question than any holdout. The holdout asks whether the rule survived history it was shown not to fit; forward asks whether it survives after the rule existed and you still knew it. That is the harder question, and it is the one that decides whether to size up — a contrast spelled out in backtest vs live trading.
Sample size has to be honest in both directions. A holdout of three months cannot condemn a strategy any more than it can bless one, and forward data accumulates slower than either — a handful of trades proves nothing and a smooth month is not a verdict. This is why the forward track reports against expectation rather than against zero: the backtest stated what should happen, the forward measures what did, and the distance between them is the only comparison the sample can support.
How to check a holdout protocol
Before trusting any out-of-sample number, answer these:
- Are the slice boundaries stored as dates stamped at registration, or as percentages that move when data is added?
- Is the rule stored as a hash, and did it stay identical between registration and verdict?
- Was the holdout opened once, with that opening recorded as the verdict run?
- Do the joints carry a purge strip at least as long as the holding horizon, plus an embargo after it?
- Were fold parameters chosen per fold on training data, or fixed in advance — and is a fold set drawn by eye labeled a stability scan?
- Is the report metric read net of costs, with the trial-adjusted threshold next to it?
- Is a forward track running on data that did not exist at registration?
Normal Trading registers the graph once, stamps train, validation, holdout and forward as dates, and keeps the sealed window sealed until the verdict. You can watch the stamp without opening the holdout on a replay at /chart.