Education

Backtest vs Live Trading — Which Part of the Gap Was Already in the Test

You went live and the curve died. The usual suspects in search results are slippage and psychology, and both are real, but neither is a diagnosis. The gap between a backtest and a live account is at least two different problems wearing one name, and the fix depends on which half you are in.

Half one: the backtest was lying — same-bar fills, unpaid costs, a Sharpe that was the best of many attempts. Half two: the backtest was honest and live still differs, because execution, regime and capacity are not fully modellable. A judged backtest already removes most of the first half, which is why the useful question is not "does a gap exist" but "which parts of my gap were already in the test".

Splitting the gap before blaming it

Every component below either lives inside the test or outside it. Treating them as one number is why traders keep buying better data when their problem was a fill model.

Part of the gapWhat it isHandled by a judged test?
Same-bar fillOrder filled at the price of the bar that generated the signalYes — replaced by the next bar's open
Unpaid costsCommission, spread and slippage missing from the curveYes — a cost profile on every fill, drag printed
Lucky-N SharpeBest result of many attempts presented as the expectationYes — the threshold moves with the trial count
Latency and routingLive order reaches the venue later than the test assumedNo
Spread regimeYour trading hours are wider than the historical modelPartly — modelled spread, optimistic profiles flagged
CapacityYour size consumes more than the bar's share of volumePartly — volume cap with partial fills
RegimeThe live period behaves unlike the test periodNo — only a forward track tests this
Your own executionHesitation, discretionary overrides, size changesNo

The first three rows are the reason a judged test exists. If your backtest still fills at the close of the signal bar, pays no commission, and reports the best of forty parameter sets, the gap you see live was manufactured on the chart, not in the market.

What an honest test already contains

Three corrections are cheap and each removes a whole category of surprise.

Next-bar fill. The signal forms on bar i, the order is sent afterwards, and it fills at the open of bar i+1, plus modelled slippage. You lose the closing price of the signal bar, and that loss is the honest price of not time-travelling. Live trading still adds latency on top of that — but now the live difference is an increment, not a chasm.

Costs on every fill. Commission, half the spread on the wrong side, and a slippage term are applied per trade, and the report prints how much of the year they ate as cost_drag_pct. A strategy that fails only when costs are on was never profitable; it was free to trade in the test.

Thresholds that follow N. A Sharpe read without the number of attempts behind it is a label, not a measurement. The bar you must clear is recomputed from the trial count (how many backtests is too many), and the same story is told in what the Sharpe ratio measures.

What is left after the test stops lying

Honest fill logic is still a model, and three residues survive it.

Fill quality. Next-bar open is an approximation of a live market order, not a copy of one: the venue sees your order a few milliseconds later, the touch has moved, and on fast instruments that movement is a real cost the test cannot price. Gaps through stops behave the same way. A stop resting at 100 when the market opens at 94 is filled at 94, not at 100 — the test is honest about this, and live trading is honest about it too, which is why the loss looks identical and yet still surprises people.

Size and capacity. A bar has so much volume, and an order may not consume more than a fixed share of it — 10% of a daily bar, 1% of a minute bar, as a working default. Beyond that, fills spill into later bars or cancel. Your live account hits this earlier than you think because you are not the only participant taking the same liquidity.

Regime. A backtest, however judged, replays one history. If the live period opens in a different volatility regime, no protocol inside the test could have known. Only new data settles this, which is what the forward track in out-of-sample testing is for.

Worked example — arithmetic, not a measurement. Suppose a tested strategy makes 120 trades a year and you attribute a 10 percentage-point annual gap to causes: 120 trades × 3.5 bp of extra live fill cost is 4.2 points; wider spreads in your session, 2.8 points; one adverse regime stretch, 3 points. The decomposition sums to the gap and tells you to fix fills first, because it is the only term you control. The point is the method: split the gap into priced components before you invent a narrative.

A published independent observation

The strongest public evidence that pretty backtests leak comes from outside this industry's marketing. A published independent 30-day test of a popular portfolio-trading service reported that its backtests promised annualized returns of 11.4%, 13.1% and 16.8%, while the live account lagged by 8 to 15 percentage points a year, and the study named the cause directly: fills happened at prices the backtest had assumed. Treat that as a dated third-party observation, not as a benchmark for any platform including this one — its value is the mechanism it documents, and the number should be re-checked against the original before anyone quotes it.

The lesson transfers regardless of the exact figure: the dominant term in the gap is usually the price at which the test assumed you traded. That is a fill-model problem, and it belongs to the first half of the gap — the half a judged backtest removes before you ever go live.

Paper trading is neither live nor a holdout

Paper on a live feed looks like a compromise between backtest and reality, and it is, but it does not buy the property people think it buys. The feed is live; the process is not sealed. You can still re-tune after a bad week, you still know the rule intimately, and the simulator still models fills and queue position its own way. Running paper right after tuning is not a holdout, it is more in-sample time wearing a live price feed, with none of the seal described in out-of-sample testing.

Paper earns its keep only after registration: freeze the rule, run it untouched on paper, and compare the result against the expectation the backtest printed. That comparison has a name in our reports — forward versus expectation — and it survives because nothing was changed while it accumulated.

What this platform will not close

Normal Trading is a research lab, not an execution venue, and saying so precisely is part of the product. Routing decisions, order-book placement, queue position, partial fills at the touch, and exchange-side priority are outside a bar replay by design. We model fills at the next bar's open with a cost profile, cap order size as a share of bar volume, and flag optimistic settings in the report — after that, execution quality is your broker's job.

If your edge lives in the order book — sub-second routing, queue priority, market-making — no bar platform closes that gap, and the right move is an execution environment with a broker or a code-first research stack. If your edge is a rule on bars, the lab is the right room, and the gap that matters is the one the judged test already priced.

How to read the gap after you go live

  1. Compare live fills against the test's assumed fills trade by trade. The difference is your real execution cost, in basis points.
  2. Check whether the tested number was net of costs and read next to a trial-adjusted threshold. If not, you did not have a backtest, you had a demo.
  3. Confirm stop behaviour matches: a gap through a stop fills at the next open, in the test and in your account alike.
  4. Ask whether your size exceeds the volume share the test allowed. If it does, the tested strategy was a smaller strategy.
  5. Only then consider regime. It is the only component you cannot price, and the only one worth waiting on.

If the judged backtest already fails, going live to "see what happens" buys you an expensive version of the same answer — and the drawdown it costs you is measured on the net curve (max drawdown explained). If it passes, the next honest step is a forward track on frozen rules, which you can start with a single registered replay at /chart.