How to tell whether your edge is fake

There are three data sets, not two

People talk about in-sample and out-of-sample as though there are two categories. There are three, and the third is the one almost nobody actually keeps.

1. In-sample. The data you fitted on. Every threshold, filter, and lookback was chosen while looking at it. Results here are a debugging check that the code runs, not evidence about the future.
2. Out-of-sample. Data withheld during fitting, used to check whether the fit generalises. The instant you look at it and go back to change something, it has become in-sample. Most researchers do this five or six times without noticing.
3. Held-out final test. A block of data written down in advance, never touched, evaluated exactly once. One look, ever. If the result is bad, the project dies, it does not get revised.

The gap between the second and the third is where most self-deception lives. An out-of-sample set that you have consulted repeatedly is not out-of-sample anymore, it is just in-sample data you feel virtuous about. The honest question is not did you hold data back, it is how many times did you look at the held-back data before you were happy. Write that number down. If it is more than one, your out-of-sample result carries a multiple-testing penalty exactly like the one described below.

Walk-forward analysis is a structural fix for the first problem and is covered separately in the walk-forward testing guide. This guide is about the second problem: how to detect that a result which passed a clean test is still noise.

Parameter sensitivity: the plateau and the spike

A real effect is broad. If a 40-bar lookback works and a 39-bar lookback does not, you have not found a market property, you have found a coincidence in a specific price series. The single cheapest overfitting test is to look at the neighbourhood of your chosen parameters rather than the parameters themselves.

Take a strategy with two knobs, a lookback length and an ATR stop multiple, and print out-of-sample expectancy in R for every combination. Here is what a result worth keeping looks like:

Every cell is positive, the surface changes smoothly, and the best cell is barely better than its neighbours. Nothing about that grid depends on picking the right number. You could be wrong about the lookback by 20 bars and still have roughly the same system.

Here is the same grid for a strategy that should be thrown away:

The headline number is far better: +0.31R against +0.12R. It is also completely isolated. Its immediate neighbours average slightly below zero, which means the strategy is a coin flip everywhere except one cell, and that cell is the one you would have reported. A parameter spike is not a strong result, it is a measurement of how much noise your data set contains.

Plateau ratio = mean(8 surrounding cells) ÷ peak cell Good grid: +0.108 ÷ +0.12 = 0.90 Bad grid: −0.010 ÷ +0.31 = −0.03 Rule of thumb: below about 0.6, treat the peak as noise.

Report the plateau, not the peak. If you are going to trade a configuration, trade the centre of the stable region rather than the best cell, because the best cell is the one most likely to have been lifted by luck. When a strategy has more than two knobs, print two-dimensional slices through the chosen point rather than trying to eyeball a five-dimensional surface.

Why testing 200 variants guarantees a winner

This is the piece of arithmetic that ends most backtests, and it is almost never shown. Suppose every variant you test has a true edge of exactly zero. Their measured Sharpe ratios are still scattered around zero by sampling noise. Take the best of the batch and you have not found skill, you have found the right tail of a distribution you generated yourself.

The expected maximum of N independent standard normal draws is approximately the square root of twice the natural log of N. Multiply that by the standard error of a Sharpe estimate and you get the number you should expect to see from pure noise:

SE(Sharpe) ≈ √(1 ÷ years of data) E[best of N] ≈ SE(Sharpe) × √(2 · ln N) Worked, 5 years of data, 200 variants tested: SE = √(1 ÷ 5) = 0.447 √(2·ln 200) = √(2 × 5.298) = 3.256 E[best] = 0.447 × 3.256 = 1.46 A 1.46 Sharpe is the EXPECTED winner when every variant is worthless.

Read the bottom row again. Run a grid search over a thousand configurations on five years of data and a Sharpe of 1.66 is the average outcome of testing nothing but garbage. Most retail backtesting projects sail past a thousand configurations without counting, because the count is not the number of ideas you wrote up, it is the number of configurations the search actually evaluated.

Configurations searched = (values per parameter) ^ (number of parameters) 5 parameters × 6 candidate values each = 6^5 = 7,776 configurations You tested 7,776 hypotheses, not one.

The correction intuition is Bonferroni's: to keep a 5% false-positive rate across N tests, each individual test has to clear a threshold of 0.05 ÷ N. At N = 200 that is a p-value of 0.00025, which is roughly 3.5 standard errors rather than the 1.65 a single one-sided test needs. A t-statistic of 2 on the best of 200 variants is not weak evidence, it is evidence of nothing at all.

Bonferroni is deliberately conservative and your variants are correlated rather than independent, so the true penalty sits somewhere between no correction and the full one. That is an argument for estimating the penalty, not for skipping it. The practical version: keep a log of every configuration evaluated across the whole project, and treat that running total as N.

Deflated Sharpe and the haircut

Once you have an expected best-under-the-null, you can subtract it. That is the whole idea behind deflated Sharpe ratios: compare the observed result against the noise ceiling produced by your own search rather than against zero.

Deflated z = (observed Sharpe − E[best of N]) ÷ SE(Sharpe) Observed 2.00, N = 200, 5 years: z = (2.00 − 1.46) ÷ 0.447 = 1.21 confidence ≈ 89% Naive reading of the same number: z = 2.00 ÷ 0.447 = 4.47, p ≈ 0.000004 Effective Sharpe after haircut = 2.00 − 1.46 = 0.54

The naive calculation says one in 250,000. The deflated one says about nine in ten, and leaves you an effective Sharpe of 0.54 after a 73% haircut. Both numbers come from the same backtest. The only difference is whether the search that produced it was accounted for.

Apply the haircut before anything downstream. Position sizing, expected drawdown, and time-to-recovery all scale off the edge estimate, so an unhaircut Sharpe quietly corrupts every one of them. The drawdown recovery math gets far less comfortable when the input edge is 0.54 rather than 2.00, and Monte Carlo work built on the inflated figure will understate the depth and duration of the bad stretch. There is a simulator on the risk toolkit if you want to see the difference.

The random-entry benchmark

This is the most useful control available and the least frequently run. Your strategy is a bundle of decisions: what to trade, when to enter, how long to hold, how much to risk, when to exit. Most of the measured return usually comes from the bundle, not from the signal. The random-entry benchmark separates them.

The construction is deliberately unglamorous. Keep everything the strategy does except the entry timing, then replace the entry with a random date:

1. Match the universe. Same symbols, same listing history, same liquidity filters.
2. Match the calendar window. Same start and end date, so the random arm eats the same bull and bear regimes.
3. Match the trade count. If the strategy produced 4,933 trades, generate 4,933 random entries, not 500.
4. Match the holding period distribution, not just the mean. Sample from the strategy's actual holding times.
5. Match the risk per trade and the exit logic, including stops, so results are in the same R units.
6. Randomise only the entry timing, then repeat the whole run 1,000 times to get a distribution rather than one number.

What comes back is a null distribution for your own system. If the strategy's expectancy sits inside that distribution, the signal added nothing measurable and you have been paying commissions for a random number generator wearing a chart pattern.

Here is that test run against my own research, which is the reason this section exists:

Signal contribution = strategy expectancy − matched random-entry expectancy AVWAP-reclaim swing setup, 129 symbols, 10-year universe: strategy expectancy = +0.117R over 4,933 trades 95% CI = [+0.057R, +0.174R] matched random entry = +0.086R ------------------------------------------------ signal contribution = +0.030R, CI crosses zero

The headline figure is +0.117R over 4,933 trades with a confidence interval that clears zero, which looks like a result. A risk-matched random entry on the same universe, same window, same holding periods, and same risk already captures +0.086R of it. The signal's own contribution is +0.030R, with a confidence interval that crosses zero, so this is not a validated edge. It is a stop-loss and holding-period policy applied to a long-biased universe during a period when that universe went up, plus a signal whose marginal contribution cannot be distinguished from noise at this sample size.

That is the honest reading of my best surviving setup, and it is worth more than a flattering one. An earlier and much smaller sample from the same project looked considerably better and did not survive re-testing, which is the ordinary fate of results built on a hundred-ish trades.

SE(expectancy) = σ_R ÷ √n σ_R ≈ 2.10R (R outcomes are fat-tailed) n = 4,933 SE = 2.10 ÷ 70.2 = 0.0299R 95% CI = 0.117 ± 1.96 × 0.0299 = 0.117 ± 0.059 To halve that interval you need roughly 4× the trades: ~19,700.

That last line is the part people skip. Precision improves with the square root of sample size, so the sample needed to resolve a 0.03R difference is enormous. If a small edge is what you have, you need patience and volume, and you should be running paper trading with a hard promotion gate rather than sizing up on a backtest.

Degrees of freedom against sample size

Every tunable choice is a degree of freedom, and the count is always larger than people admit. It includes the obvious knobs, but also the universe filter, the holding cap, the regime switch you added and removed, the two indicators you tried and abandoned, and the start date you nudged because 2008 looked ugly.

Trades per free parameter = out-of-sample trades ÷ free parameters 240 OOS trades ÷ 6 free parameters = 40 per parameter Working floors: under 30 per parameter → fitted, discard 30 to 100 per parameter → suggestive at best over 100 per parameter → worth a held-out final test

Those thresholds are conventions, not laws, and they get stricter as the effect size shrinks. A system claiming +1.0R expectancy needs far fewer trades to prove itself than one claiming +0.05R, because the signal-to-noise ratio is twenty times better. Sizing everything in R rather than dollars is what makes these comparisons possible in the first place, which is covered in the R-multiple and expectancy guide.

The cleanest way to reduce degrees of freedom is to delete parameters rather than optimise them. Fixing a stop at 2× ATR because that is a reasonable prior, and never tuning it, costs a little performance and removes an entire axis of overfitting. Two-parameter systems with three hundred trades routinely survive tests that nine-parameter systems with forty trades cannot.

The universe is a parameter too

Universe selection is where data snooping hides most successfully, because it does not feel like a parameter. Every one of these is a fitted choice: which symbols, which index membership date, which liquidity floor, which sectors, which start year.

• Survivorship. A universe built from today's listings has silently deleted every company that went to zero. Long-biased results against a survivor universe are inflated, sometimes severely.
• Index reconstitution. Using today's index membership across a ten-year history means you were holding the winners before they were added. That is lookahead wearing a respectable suit.
• Symbol cherry-picking. Dropping the four names that ruined the equity curve is overfitting even when each deletion had a story attached.
• Start-date selection. Beginning the test in 2010 rather than 2007 is a parameter choice worth roughly one full market regime.
• Regime coverage. A window containing one bear market gives you one observation of bear-market behaviour, no matter how many trades it holds.

State how the universe was constructed before showing any result. If the answer is that it grew organically while you worked, that is a real finding about the result's reliability, and it should be written into the report rather than left out of it. The same applies to how the strategy's own screening rules narrow the tradable set.

An honest robustness protocol

End to end, in order, with the destructive steps taken before the flattering ones:

1. Pre-register. Before touching data, write down the hypothesis, the parameters you will tune, the ranges, the success threshold, and the held-out block. Date the file.
2. Build the universe first, including delisted tickers, and freeze it. Document how it was constructed.
3. Fit in-sample only. Do not look at anything else. Count and log every configuration evaluated.
4. Print the parameter surface. Reject spikes outright. Choose the centre of the plateau, not the peak.
5. Run walk-forward so every reported trade came from parameters chosen without seeing it.
6. Run the random-entry benchmark at matched count, holding period, risk, universe, and window, 1,000 iterations. Report the strategy's percentile within that null distribution.
7. Apply the multiple-testing haircut using your logged configuration count, not the number of ideas you liked.
8. Check degrees of freedom against trade count and state trades per parameter explicitly.
9. Add realistic costs: slippage, commission, financing, partial fills, borrow. Then re-check that the edge survives.
10. Run the held-out final test once. Accept the answer. If it fails, the strategy is dead, and revising it converts the held-out block into training data forever.
11. Paper trade to a pre-committed gate before any real capital, because execution reality is a separate test from statistical reality.
12. Publish the failures alongside the survivor. A research file with no rejected variants is a sales page.

Why people fool themselves

None of this is difficult arithmetic. It gets skipped for reasons that are emotional rather than technical, and knowing the mechanism is most of the defence.

• Sunk cost. After three weeks of work, a test that says no is expensive. The cheapest moment to kill a strategy is before you have invested in it, which is exactly when nobody wants to run the test.
• Asymmetric scepticism. A bad result gets audited for bugs. A good result gets accepted. That single asymmetry is enough to manufacture an edge out of nothing over a long enough project.
• The narrative fits. Once you have a story for why the signal should work, contradicting evidence reads as a data problem rather than an answer.
• Stopping when you win. Testing continues until a result is good, then stops. That is optional stopping, and it inflates every statistic it touches.
• Identity. If being a person who has an edge matters to you, then testing the edge is testing yourself, and you will unconsciously build a test that is easy to pass.

The structural fix is to make the decision before you can see the answer. Pre-register the threshold, define what failure looks like, and hand the kill decision to a rule instead of to yourself in a moment of enthusiasm. The same discipline applies once real trades exist: log the planned edge, then compare it against realised R in a trade journal that computes the number for you, and read the result the way you would read someone else's.

Two more things that help. Write the report as though a hostile reviewer will read it, which is what reading a backtest report is designed to train, and keep every negative result, because a rejected variant is a permanent asset that stops you paying to rediscover the same dead end. My own engine and the tests that failed on it are documented on the swing screener page.

Nothing here is a route to a validated edge, and no live account performance is claimed anywhere on this site. Backtests, walk-forward results, and random-entry benchmarks describe historical simulations only. Real trading adds slippage, gaps, outages, borrow constraints, and your own behaviour under pressure, all of which make live results worse than tested ones. Trading involves substantial risk of loss.

Educational content, not financial advice. No live profit-and-loss figures are claimed anywhere on this site; backtest and walk-forward results are always labelled as such. Full terms: /terms.html

Tools referenced in this guide

• Swing Screener — the no-lookahead backtest engine, the walk-forward runs, and the negative results that came out of them.
• Trader's risk toolkit — Monte Carlo and expectancy calculators for stress-testing an edge estimate after the haircut.
• Walk-forward testing guide — the structural side of honest testing: lookahead bias, rolling windows, and survivorship.