Reading a backtest report honestly

The filter that ran before you saw it

Before any statistic matters, ask a question the report itself cannot answer: would this report still be in front of me if it had come out badly? Almost always the answer is no, and that single fact contaminates everything below it.

A vendor who tests 200 parameter combinations and publishes the best one is not showing you a strategy result. They are showing you the maximum of 200 draws, which is a very different object with a very different expected out-of-sample value. A screenshot in a Discord is worse: nobody screenshots the flat quarter, so what circulates is the right tail of a population you never get to see.

This applies to your own work with no discount. If you stopped iterating the moment the equity curve looked good, you ran the same selection process on yourself, just without keeping the log. That is the mechanism behind backtest overfitting, and it is why the number of configurations you tried is a more important disclosure than any statistic in the table.

The whole guide in one table

Win rate is not a result

Win rate is the most quoted and least informative number in retail trading. On its own it says nothing about whether a system makes money, because it does not describe how big the winners are relative to the losers.

The break-even win rate falls straight out of the payoff ratio, where payoff is average win divided by average loss, both measured in R:

payoff = average win R / average loss R break-even win % = 1 / (1 + payoff) example: payoff 2.0 -> 1 / 3.0 = 33.3%

So a headline of 68% winners means one thing on a 2:1 payoff and something close to nothing on a 0.5:1 payoff. Any report that gives you a win rate without an average win and average loss beside it has withheld the half of the pair that determines the sign of the result. Expectancy in R is the number that actually combines them, and the arithmetic is in the R-multiple and expectancy guide.

High win rates also get advertised for a behavioural reason: they are pleasant to trade and easy to sell. They are usually produced by taking profits quickly while letting losers run to the stop, which gives a long comfortable string of green followed by a cluster of full-sized losses.

Profit factor and the single best trade

Profit factor is gross profit divided by gross loss. Above 1.0 the system made money, and vendors like it because a value like 2.1 sounds sturdy. It is extremely fragile to outliers.

profit factor = sum of winning trades / |sum of losing trades| 80 trades, gross profit 42.0R, gross loss 20.0R PF = 42.0 / 20.0 = 2.10

Now remove the single best trade, which happened to be +14.0R on a biotech gap:

Always recompute profit factor and expectancy with the best trade deleted, and again with the best three deleted. If the result collapses, the strategy is not a repeatable process, it is a lottery ticket that happened to hit inside the window. That is not automatically disqualifying, since some genuine trend systems really do earn everything from a handful of trades, but then the report should say so and the position sizing should be built for it.

Run it in reverse too. If deleting the worst trade transforms the result, you have learned something about the stop logic rather than about the entry.

Means, medians, and the trade you will actually get

Average win and average loss are means, and trade distributions are skewed enough that means describe a trade nobody took. Consider 20 winners: nineteen between +0.3R and +0.8R, one at +9.0R. The mean win is about +1.0R. The median win is +0.5R. Half your winning trades will come in under half the advertised average.

Ask for the median trade, the median win, and the median loss. When the median trade in R is negative while the mean is positive, the system is tail-dependent, and tail-dependent systems feel completely different to trade: long stretches of slow bleed punctuated by rescues you cannot schedule. Losses tend to be far better behaved than wins, because the stop truncates them, so the mean and median loss usually sit close together. A large gap between mean and median loss is a red flag for gaps through stops or missing slippage assumptions.

A percentile table beats both. The 10th, 25th, 50th, 75th and 90th percentile of trade R tells you the shape in five numbers. If your own journal software cannot produce that, it is worth fixing, because the same distribution feeds every sizing decision you make. TradeLog keeps per-trade R locally so the distribution is recoverable rather than reduced to a headline average.

Sharpe ratio has an error bar too

Sharpe is a ratio of mean excess return to the standard deviation of returns. Reported without a sample size it is close to meaningless, because the Sharpe estimate itself has a standard error that shrinks only with the square root of the number of observations. A useful approximation, valid for roughly normal returns:

SE(Sharpe) ~ sqrt( (1 + S^2 / 2) / N ) S = Sharpe at the SAME frequency as the observations N = number of return observations

Work an example the way a real report presents it. Someone quotes an annualised Sharpe of 1.0 built from 40 monthly observations, a little over three years:

monthly S = 1.0 / sqrt(12) = 0.289 SE(monthly) = sqrt((1 + 0.042) / 40) = 0.161 SE(annual) = 0.161 x sqrt(12) = 0.559 95% CI on the annualised Sharpe = 1.0 +/- 1.96 x 0.559 = -0.10 to 2.10

The interval contains zero. A reported Sharpe of 1.0 on that sample is statistically indistinguishable from a strategy with no edge at all. Push the sample to 400 monthly observations, which is 33 years and more history than most tested rules have, and the standard error falls to about 0.18, giving roughly 0.65 to 1.35. Better, still not precise.

Two further inflations are worth naming. First, annualising by multiplying a daily Sharpe by the square root of 252 assumes returns are independent and identically distributed. Equity curves from trend and momentum rules are usually positively autocorrelated, and positive autocorrelation makes that scaling overstate the annual figure. Second, any strategy holding illiquid or thinly marked positions smooths its own returns, which suppresses the denominator and inflates Sharpe for reasons that have nothing to do with skill. Treat any annualised Sharpe as an upper bound unless the report states the observation frequency and the autocorrelation.

Max drawdown depends on where you measured it

Max drawdown is the largest peak-to-trough fall in equity. The question the number never answers by itself is: equity measured how?

• Closed-trade equity only updates when a position is exited, so open losses are invisible. This produces the smallest and least honest drawdown figure, and it is the default in a surprising amount of software.
• Daily mark-to-market equity includes open positions at each close. This is the minimum acceptable basis.
• Intraday equity includes the worst point reached inside the day. This is what your broker screen and your nervous system actually experience.
• Gap risk sits on top of all three. A stop at a price does not guarantee a fill at that price, and a backtest that assumes it does understates every drawdown.

The same underlying track can report a 12% closed-trade drawdown, 19% on daily marks, and 24% intraday. All three are correct. Only one of them is the number you would have had to sit through.

Then there is the deeper problem: max drawdown is a single extreme realisation, not a stable parameter. It is the worst thing that happened once. Expected maximum drawdown grows with the length of the sample, so a longer backtest will report a bigger one even if nothing about the strategy changed. The right way to use it is as an input to a resampling exercise rather than as a promise: shuffle the trade sequence a few thousand times and look at the distribution of drawdowns your own R-distribution generates. The Monte Carlo tool does exactly that, and the recovery arithmetic is in the drawdown recovery guide. The historical max is usually near the optimistic end of that distribution.

CAGR on a window someone chose

Compound annual growth rate is the most date-sensitive statistic in the table. Shift the start by six months, or end the window before rather than after a bad quarter, and the figure moves enormously. Nobody has to lie for this to happen, they only have to pick a window that felt natural.

Three questions defuse it. Was the window chosen before or after the strategy was developed? What does the year-by-year breakdown look like, rather than the single compounded number? And what did the same universe do buy-and-hold over the same window? A rule tested only on US equities from 2009 to 2021 was tested inside one of the longest equity bull markets on record, and a great deal of what looks like edge in that window is beta wearing a costume.

CAGR also embeds a compounding assumption that only holds if you could actually have scaled into it. Size that works on $30,000 may not fill at $3,000,000 in the same names, and a report that compounds without a liquidity or capacity note has quietly assumed infinite depth.

Sample size, and the confidence interval nobody prints

This is the section that decides whether the rest of the report is worth reading. An expectancy figure without an interval around it is a point estimate presented as a fact. The interval is simple to compute if the report gives you the number of trades and the standard deviation of trade R:

SE(mean R) = s / sqrt(N) 95% CI = mean R +/- 1.96 x SE s = standard deviation of per-trade R N = number of trades

Worked on a typical published sample. A course sells a setup showing +0.20R expectancy over 60 trades, with a per-trade standard deviation of 1.4R, which is unremarkable for a system with a hard stop and open-ended winners:

SE = 1.4 / sqrt(60) = 0.181 95% CI = 0.20 +/- 1.96 x 0.181 = -0.154R to +0.554R

The interval covers everything from a losing system to an excellent one. The honest summary of that backtest is not that the setup makes 0.20R per trade. It is that 60 trades cannot tell you whether the setup makes money.

Because the interval narrows with the square root of N, buying precision is expensive. Halving the width takes four times the trades. The sample size needed for a target half-width w is:

N = (1.96 x s / w)^2 s = 1.4R, want half-width w = 0.10R N = (1.96 x 1.4 / 0.10)^2 = 27.44^2 ~ 753 trades same s, want w = 0.05R N = (1.96 x 1.4 / 0.05)^2 ~ 3,011 trades

Applying this to my own research

I have run this test on my own work, and it is worth showing because it does not flatter me. My walk-forward testing produced +0.117R over 4,933 trades on a 129-symbol, 10-year universe, with a 95% confidence interval of +0.057 to +0.174 for an anchored-VWAP reclaim setup. That is a large sample and the interval sits above zero. It is still not an edge, for one reason: a risk-matched random entry on the same universe already captures +0.086R, so the signal contributes about +0.030R of its own, and the confidence interval on that increment crosses zero. That is not a validated edge, and I do not claim it is one. Method notes are in the walk-forward guide and the engine is described on the swing screener page.

That is the point of the section. Nearly 5,000 trades, a decade of data, an interval that excludes zero on the raw number, and the result still fails once the control is added. If a sample that size can end up inconclusive, a 60-trade Discord screenshot is not evidence of anything.

One more caution about trade counts. Trades taken across a correlated universe on overlapping dates are not independent observations. A market-wide down week produces dozens of simultaneous losers that behave like one bet. The effective sample size is meaningfully smaller than the raw count, which means every interval above is, if anything, too narrow.

Survivorship in the symbol universe

Ask where the symbol list came from. If it is today's index membership, or today's liquid names, or a watchlist someone maintains, then every company that delisted, went to zero, was acquired at a discount, or simply became too small to notice has already been removed. The universe is a set of survivors, and testing entry rules on survivors makes almost any long-biased rule look better than it was.

A clean test needs point-in-time constituent data and delisting returns, so a bankruptcy shows up as the loss it was rather than as a gap in the file. Very little retail-accessible data has this. My own 129-symbol universe was assembled with hindsight and I state that as a limitation rather than pretending otherwise, which is part of why I weight the random-entry control more heavily than the headline expectancy.

Survivorship also hides in the strategy definition. Rules containing conditions like liquid enough to trade or has a real product are hindsight filters if the liquidity or the product only became visible later.

Look-ahead leaks, in order of how often I find them

• Same-bar signal and fill. A signal computed from a bar's close and filled at that same close. Fill on the next bar instead.
• Whole-sample normalisation. Z-scoring, volatility targeting, or picking a threshold using statistics from the entire history. The future leaks into the past. Fit every transform on a trailing window only.
• Restated fundamentals. Earnings and index memberships get revised later. Using today's value on a historical date gives the rule information nobody had.
• Split and dividend adjustment. Adjusted prices are right for returns and wrong for anything price-level based, such as a fixed dollar filter. Unadjusted series break the other way.
• Stop and target inside one bar. When both sit inside the same bar's range, the backtest has to guess the order. Assuming the target filled first is a leak. Assume the stop.
• Parameter selection on the test set. Choosing the lookback or the threshold from the full-sample result, then reporting that result as if it were out of sample.
• Universe selection after the fact. Picking symbols known to have trended, then testing a trend rule on them.

The smell test for leakage is counterintuitive: results that are too clean are the warning sign, not the reassurance. A high win rate combined with a small drawdown and an equity curve that goes up in a near-straight line is far more likely to be a leak than a discovery. Real edges are noisy, and their equity curves have flat stretches that last uncomfortably long. When I see a suspiciously smooth curve I go looking for the leak before I look at the returns, and it usually turns out to be normalisation or same-bar fills. The structural fix is walk-forward rather than a single split, which is covered in the walk-forward testing guide.

Costs, and why a penny matters more than you think

Most reports either omit costs or apply a flat commission and stop there. The full set is commission, half the bid-ask spread on each side, slippage against the quoted price, borrow cost on shorts, and financing on anything held with leverage. Expressed in R, the drag is:

cost per share (round trip) = 2 x commission + spread crossed + slippage cost drag in R = cost per share / (1R per share) example: 0.005 commission x2, 0.04 spread, 0.04 slippage = 0.09 per share round trip

The critical property is that cost drag in R is inversely proportional to your stop distance. The same nine cents is trivial on a wide swing stop and fatal on a tight intraday one:

A single cent per share of unmodelled slippage is 0.04R on a 25 cent stop and 0.002R on a five dollar stop. This is the mechanism by which most short-timeframe backtests die: the gross edge is real and smaller than the friction. It also explains why tighter stops are not free, since the same tightness that buys you a bigger share count multiplies your cost drag. That tradeoff is the subject of the stop placement guide and the sizing side is in position sizing per trade.

Borrow deserves its own line. Short backtests routinely assume you could borrow the shares, at a normal fee, on the day you wanted them. For the exact names that fall hardest, borrow is often unavailable or costs several percent annualised, and that cost lands on the trades that made the report look good. Leveraged and inverse products carry their own structural drag on top, described in the leveraged ETF decay guide.

The benchmark and the random-entry control

A strategy result on its own is not a claim. It becomes a claim only next to what you would have got without it. Two comparisons are non-negotiable.

1. Buy and hold the same universe over the same window, risk-adjusted. If the rule returns less per unit of drawdown than owning the basket, the rule is worse than doing nothing, whatever the CAGR says.
2. A risk-matched random-entry control. Same symbols, same number of trades, same holding-period distribution, same stop and target geometry, entry dates chosen at random. This isolates how much of the result came from the signal rather than from the universe, the era, and the exit rules.

The random-entry control is the one almost nobody runs, and it is the most informative test in this guide. Exits, stops and universe selection do a great deal of the work in most systems, and a random-entry baseline reveals how much. In my own case it consumed roughly three quarters of the measured expectancy, which is why I describe the remainder as unvalidated rather than as an edge.

The last step before any of this touches money is a forward period the model has never seen, executed with real order handling. Backtest to forward test to small live size, in that order, is covered in paper to live trading.

Read it in this order

1. Who produced it, and how many configurations were tried. No answer means treat the report as an advertisement.
2. Sample size and trade overlap. Below a few hundred trades, or with heavy date overlap, say so and stop.
3. Confidence interval on expectancy. Compute it from N and the standard deviation of R. If it crosses zero, nothing further matters.
4. Costs. Convert commission, spread and slippage to R using the typical stop distance, then subtract. Unstated means omitted.
5. Delete the best trade, then the best three, and recompute profit factor and expectancy.
6. Mean versus median for trade R, wins and losses. A negative median with a positive mean is a tail-dependent system.
7. Drawdown basis. Closed-trade, daily mark, or intraday. Then resample the sequence for the distribution, not the one draw.
8. Look-ahead audit. Same-bar fills, whole-sample normalisation, restated data, intra-bar ordering. Smooth curves get audited hardest.
9. Universe construction. Point-in-time membership and delisting returns, or an explicit survivorship caveat.
10. Benchmark and random-entry control. If neither is present, the report has not made a claim yet.
11. Only then the equity curve, the part designed to persuade you and the part carrying the least information.

None of this makes a strategy safe. Passing every check above means a result is not obviously broken, which is a much weaker statement than saying it will work. Markets change, samples end, and every figure discussed here is a measurement of the past. Trading involves substantial risk of loss, backtested and hypothetical results have well-documented limitations, and nothing on this page is a recommendation to trade anything.

Educational content, not financial advice. No live profit-and-loss figures are claimed anywhere on this site; backtest and walk-forward results are always labelled as such. Full terms: /terms.html

Tools referenced in this guide

• Trader's risk toolkit — expectancy and Monte Carlo tools for putting an interval around a backtest result.
• TradeLog — per-trade R kept locally, so distributions and medians stay recoverable rather than averaged away.
• Walk-forward testing guide — the testing structure that produces a report worth auditing in the first place.