How backtests lie, and what an honest one looks like

What is lookahead bias?

Lookahead bias is using information at bar t that was not available until after bar t. It is almost never deliberate, and it is almost always fatal to the result.

The classic version is off-by-one indexing: computing a signal from the daily close and then filling the order at that same close. In reality you cannot know the close until the session is over, so the fill must land on the next bar's open. One index shift can turn a losing system into a spectacular one.

WRONG RIGHT signal = close[t] .gt. sma[t] signal = close[t] .gt. sma[t] fill = close[t] fill = open[t+1] The signal is knowable only AFTER close[t] prints, so the earliest possible fill is the next bar's open. One-line version of the test: shift every signal forward by one bar and re-run. If the edge disappears, the edge WAS the shift.

That last check is the cheapest audit in quantitative research and almost nobody runs it. Shift the signal series forward by one bar and re-run the whole test. A real effect degrades a little. A lookahead artefact evaporates entirely, because the only thing it was measuring was the future.

The bias catalogue, with the code smell for each

Biases are easier to hunt when you know what the defect looks like in source rather than what it is called in a textbook. Each row below is a specific thing to grep your own code for.

Survivorship bias in more detail

Survivorship bias is testing on a universe that only contains what is still listed today. Delisted, acquired, and bankrupt tickers vanish from most convenient datasets, so the test never has to hold the ones that went to zero. Any long-biased strategy looks better against a universe that quietly deleted its worst outcomes.

The fix is a point-in-time universe: for each date, the list of names that were actually available and actually satisfied your filters on that date. If you cannot construct one, the honest mitigation is to say so, restrict to large, liquid names where delisting risk is lower, and treat the result as an upper bound rather than an estimate. Reporting the limitation is not a weakness in a write-up; omitting it is.

In-sample, out-of-sample, walk-forward

Three levels of honesty:

Walk-forward works like this: fit parameters on a training window, test on the block immediately after it, then slide both windows forward and repeat. The reported result is the concatenation of every out-of-sample block. Because every trade in the result came from parameters chosen without seeing that trade, the equity curve is at least structurally honest.

The word "structurally" is doing work there. Walk-forward fixes the ordering problem. It does not fix the multiple-testing problem, because you can run the whole walk-forward procedure forty times with forty different ideas and report the best one. That failure mode has its own guide: backtest overfitting.

Anchored versus rolling walk-forward

There are two ways to slide the training window and they encode different beliefs about the market. Rolling keeps the training window a fixed length and drops the oldest data as it advances; it assumes the recent past is more relevant than the distant past. Anchored keeps the start date fixed and grows the training window; it assumes all history is informative and prefers a larger sample.

A concrete schedule over ten years of daily data, training on 36 months, testing on the following 6, stepping 6 months at a time:

folds = (120 months − 36 months) ÷ 6 = 14 folds out-of-sample coverage = 14 × 6 = 84 months = 7.0 years fold 1 rolling: train m1–m36 anchored: train m1–m36 test m37–m42 fold 2 rolling: train m7–m42 anchored: train m1–m42 test m43–m48 fold 3 rolling: train m13–m48 anchored: train m1–m48 test m49–m54 ... fold 14 rolling: train m79–m114 anchored: train m1–m114 test m115–m120 Rolling training window: always 36 months. Anchored training window: grows 36 → 114 months.

Two rules that matter more than the choice itself. First, decide which one you are using before you see any results, because trying both and reporting the better one is a two-trial experiment reported as one. Second, insert a gap between the training window and the test window equal to your maximum holding period, otherwise a position opened at the end of training resolves inside the test block and quietly contaminates it.

Parameters versus trades: the overfitting arithmetic

Every tunable parameter is a degree of freedom you spend fitting noise. The rough working standard is 30 to 50 out-of-sample trades per tuned parameter, and "tuned" includes every threshold, lookback, multiplier, and filter you tried and kept — plus the ones you tried and discarded, which people never count.

The other half of the arithmetic is the size of the search space, and it grows multiplicatively rather than additively. Testing five values of each parameter:

combinations = 5^k for k parameters 2 params → 25 combinations 3 params → 125 4 params → 625 5 params → 3,125 6 params → 15,625 At a 5% false-positive rate, the number of combinations that look 'significant' purely by chance: 2 params → 1.2 4 params → 31.2 6 params → 781.2

A six-parameter grid search produces roughly 781 spurious winners before any real effect exists. You are not choosing the best parameters; you are choosing the luckiest ones. This is why parameter-sensitivity surfaces matter more than the winning cell — a real effect shows a broad plateau, a fitted one shows an isolated spike.

Multiple testing: why trying 40 variants guarantees a winner

Run one test at a 5% significance threshold on a strategy with no edge and you have a 5% chance of a false positive. Run N independent tests and the probability that at least one comes back "significant" is 1 − 0.95^N.

Read the 40-variant row carefully, because it is the realistic case. If you try forty ideas — and a weekend of research is easily forty ideas once you count parameter variations — there is an 87% chance at least one clears a 5% threshold on pure noise, and you should expect the best of the forty to show a t-statistic around 2.7 even if every single one is worthless. A t-stat of 2.7 looks convincing. In that context it is the expected value of the maximum, not evidence.

The corrections are unpleasant and that is the point. The Bonferroni threshold divides your significance level by the number of trials: at 40 variants you need p below 0.00125 rather than 0.05, which for a single-tailed test is roughly a t-statistic of 3.0. The alternative — and the one that actually works in practice — is to count and publish the number of variants tried, so a reader can apply the discount themselves. A write-up that does not state N is not reporting a result, it is reporting a maximum.

The cost model, and how fast it eats an edge

Slippage is not a rounding error at the scale of edges that survive honest testing. Express it in R and the damage becomes obvious. Take a $48 entry with a stop at $45.60, so 1R equals $2.40 per share, and apply a fixed slippage per side against a gross edge of +0.117R:

round-trip cost in R = 2 × slippage per side ÷ 1R per share 2 × $0.05 ÷ $2.40 = 0.0417R +0.117R − 0.0417R = +0.0753R

Fifteen cents per side on a $48 stock — entirely ordinary on a fast open or a thin name — turns a positive edge negative. And that is measured against the gross figure. Measured against the portion of the edge actually attributable to the signal, discussed below at roughly +0.030R, two cents per side consumes 56% of it and five cents consumes more than all of it.

• Slippage — assume worse fills than mid, especially on gaps, opens, and small caps. Model it as a function of spread and size, not as a constant.
• Commission and financing — small per trade, decisive at high frequency. Overnight financing on margin is a real drag on multi-day holds.
• Partial fills — a backtest fills 100% of a limit order that reality would have missed. Limit-order strategies are the most systematically overstated category there is.
• Borrow availability and cost for shorts, which the data will not tell you about, and which is worst exactly on the names a short strategy wants.
• Your own behaviour — the backtest takes every signal; you will not, and the ones you skip will not be a random sample.

Confidence intervals on expectancy

A backtest result is a sample mean and should never be reported without its uncertainty. The interval is the standard one:

SE = σ_R ÷ √N 95% CI = mean ± 1.96 × SE Worked, on the anchored-VWAP walk-forward result: mean = +0.117R, N = 4,933, reported CI = [+0.057, +0.174] half-width = (0.174 − 0.057) ÷ 2 = 0.0585 √4,933 = 70.235 implied σ_R = 0.0585 × 70.235 ÷ 1.96 = 2.10 check: 1.96 × 2.10 ÷ 70.235 = 0.0585 OK

Reverse-engineering σ from a published interval is a useful habit, because it tells you whether the interval is even self-consistent. An R standard deviation of 2.10 is plausible for a swing system with a long right tail. If the arithmetic had implied σ = 0.3, the interval would be reporting a distribution nobody has ever traded.

Turn the same formula around and it tells you how much data an edge of a given size actually needs — which is the number that ends most retail backtest arguments:

The random-entry control

This is the benchmark almost nobody runs and it is the one that changes conclusions. The question a backtest is implicitly claiming to answer is "does my signal work?" But a profitable result can come from four sources: the signal, the exit rules, the position sizing, or simply being long a universe that drifted up. A random-entry control separates them.

1. Keep everything except the entry trigger: same universe, same date range, same stop rule, same exit rule, same holding period, same position sizing.
2. Replace the signal with a random entry, risk-matched so the average risk per trade and the number of trades match the real system.
3. Run it many times and take the distribution, not one run.
4. Compare. The real result minus the random-entry result is the only part of the edge your signal can claim.

On my own anchored-VWAP work, that control is the reason the honest conclusion is negative even though the headline is positive:

AVWAP reclaim, walk-forward: +0.117R over 4,933 trades 129 symbols, 10-year universe 95% CI [+0.057, +0.174] Risk-matched random entry: +0.086R (73% of the headline) Signal's own contribution: 0.117 − 0.086 = +0.030R SE of the difference = 2.10 × √(2 ÷ 4,933) = 0.042 95% CI on the difference = [−0.052, +0.114] That interval crosses zero. The signal is NOT a validated edge.

Nearly three quarters of the headline number was available to a coin flip using the same exits, the same universe, and the same risk management. Whatever is working there is mostly the exit discipline and the drift of the universe, not the trigger. And the residual +0.030R would need roughly 18,758 trades to distinguish from zero at 95% confidence, so it is not something a larger sample of the same design is going to rescue.

What my own testing produced

The outcome was mostly negative, which is the normal outcome and the reason the exercise is worth doing at all.

One figure needs an explicit label. An earlier version of the anchored-VWAP result, run on a hand-picked handful of names over barely a hundred trades, came back at roughly twice the figure above. That number is retired: it failed my own adversarial re-test, it did not reproduce on a wider universe, and I no longer republish it. It stands here only as an example of what a small sample plus an unstated number of variants produces — a hundred-odd trades, per the table above, is a fraction of the 1,233 needed to distinguish even the smaller +0.117R figure from zero.

The two rejected filters are worth as much as the survivor. Both were ideas I was confident about before testing; both cut expectancy on the same data and were dropped rather than quietly retuned. An honest test is one that is allowed to say no. Full write-up and engine →

Reading someone else's backtest

A checklist you can run in about two minutes, ordered so that the earliest failure saves you the rest:

1. How many out-of-sample trades? Under 100, stop reading. Under 30, it is a story about eight trades.
2. Is there a confidence interval? A bare point estimate is a claim, not a measurement. If there is no interval, compute one from the trade count and a plausible σ before believing anything.
3. How many variants were tried? If the number is not stated, assume it is large and apply the 40-variant discount from the table above.
4. What are the assumed costs? No slippage figure means the result is gross, and gross results on small edges are meaningless.
5. What is the universe, and is it point-in-time? A current constituent list means survivorship bias is baked in.
6. Where does the test period start, and why there? An unexplained start date that is not the beginning of available data is a red flag.
7. Is there a random-entry or buy-and-hold control? Without one, you cannot separate the signal from the exits from the drift.
8. How many parameters, and is there a sensitivity surface? A single winning parameter set with no neighbourhood shown is a spike, not a plateau.
9. Where are the negative results? A write-up with no failed variants is a sales page. Real research kills more ideas than it keeps.
10. Does the author state what would falsify it? If nothing could, it was never a test.

The line-by-line version of that checklist applied to a real report is in how to read a backtest report, and the trade-level metrics it produces are defined in the R-multiple and expectancy guide. Once a result survives all of this, the next problem is that surviving a backtest is not the same as surviving execution — going from paper to live covers the gap, and the promotion gate on my own execution side is 60 closed paper trades with positive realised expectancy, against which the current record is zero.

Educational content, not financial advice. No live profit-and-loss figures are claimed anywhere on this site; backtest and walk-forward results are always labelled as such. Full terms: /terms.html

Tools referenced in this guide

• Swing Screener — the no-lookahead backtest engine and walk-forward results.
• Trading bot — the execution side, with a hard gate requiring 60 profitable paper trades before any live broker.
• R-multiple & expectancy guide — how to measure what a test produced.