I ran a robustness test on my own trading strategy and it failed

2026-07-31 · Yusuf Gadelrab

The number I was proud of survived a walk-forward test, a ten-year universe, and nearly five thousand trades. Then I ran one more test, the one almost nobody runs, and it took most of the number away.

This is the writeup of that second test. I am publishing it because the first version of this result is the kind of thing people put in a pinned post, and the second version is the kind of thing people quietly delete.

What I was actually testing

I spent months building swing trading setups and then testing them properly, which in practice means testing them until they stop working. Three candidates made it far enough to be measured seriously:

• A gap and opening range breakout proxy, run on daily bars. The classic day trading structure, approximated at a resolution I could test across a decade.
• VCP, the volatility contraction pattern. This is the one I wanted to work, because the literature around it is genuinely beautiful and because it is the setup that made me want to build a screener in the first place.
• An anchored VWAP reclaim. Price loses the anchored volume weighted average price from a defined event, then takes it back.

Everything is measured in R, where 1R is the amount risked from entry to the stop. Expectancy is the average R per trade. If you want the arithmetic behind that, I wrote it out in the R-multiple and expectancy guide. The short version: expectancy is the only performance number that does not care how big your account is, which is why it is the only one I trust from a student with a small one.

Round one: walk-forward

Walk-forward testing means you fit on a window, test on the window after it, roll forward, and repeat, so the model never sees the data it is graded on. It is the minimum bar. It is not a high bar. I wrote up the mechanics and the specific ways backtests lie in the walk-forward guide.

Losing the gap proxy did not hurt. Losing VCP did. That was the setup I had read the most about, drawn the most charts of, and quietly assumed I would end up trading. Breakeven is a specific kind of insult: it is not wrong enough to be interesting and not right enough to use.

So I had one survivor, and I ran it across a proper universe: 129 symbols, ten years, 4,933 trades. Expectancy came out at +0.117R, with a 95% confidence interval of +0.057R to +0.174R. The whole interval sits above zero. An earlier version of this result came from a far smaller sample and a much prettier number, and the bigger sample cut it down. That is what bigger samples do, and I have retired the old figure everywhere it appeared.

At that point I had what most retail writeups would call an edge. Positive expectancy, large sample, tight enough interval, out-of-sample tested. I nearly stopped there.

The test almost nobody runs

Here is the question I could not put down. My strategy has two parts: a signal that says when to enter, and a risk structure that says how much to risk, where the stop goes, and when to get out. When the whole thing makes +0.117R, which part earned it?

The way to find out is to keep the risk structure and destroy the signal. Enter at random. Same universe, same position sizing, same stop distance, same exit logic, same number of trades. The only thing removed is the reason for entering.

That risk-matched random entry captured +0.086R.

A random entry with my risk rules kept roughly three quarters of the expectancy that I had been attributing to my signal.

Subtract, and the signal's own contribution is +0.030R, with a confidence interval that crosses zero. Crossing zero is the part that matters. It means the data I have cannot rule out that the signal contributes nothing at all, and that everything the system earned came from the risk management wrapped around it.

Why this is not a technicality

A confidence interval that crosses zero is not a slightly weaker result. It is a different category of result. "Probably positive, magnitude uncertain" and "cannot distinguish from nothing" are not neighbours.

And notice what the random benchmark implies about the original number. The +0.117R was real. It was measured correctly, out of sample, over a large sample. It just was not evidence of what I thought it was evidence of. Beating zero and beating a benchmark are different tests, and almost every published retail edge only runs the first one.

This is the same failure mode as a drug trial with no placebo arm. Patients got better. Nobody checked whether they would have gotten better anyway.

Two filters I was certain would help

Before the random test, I had also tried to improve the survivor with two filters that every book recommends. A market regime filter, so the system only trades when the broad market is healthy. And a new market leader filter, restricting entries to names showing leadership characteristics.

Both are intuitively obvious. Both are standard advice. Both made expectancy worse on my own data, so I removed them. That result surprised me more than the random benchmark did, and it taught me something specific: advice that is universally repeated is not thereby tested, and it is definitely not tested on your setup, your universe, and your holding period.

What I actually did about it

I did not delete the system. A +0.030R signal contribution that cannot be distinguished from zero is not proof of no edge, it is proof of insufficient evidence. Those are different, and the correct response to insufficient evidence is to gather more of it under conditions where being wrong is cheap.

So the engine has a promotion gate, and the gate lives in the code rather than in my judgment: no live broker connection until 60 closed paper trades with positive realized expectancy. There is an override environment variable, because a gate with no documented override just gets edited out at 2am, and an override that logs a warning is at least honest about what you are doing.

Current count of closed paper trades: zero. The paper engine has been running dry because the broker keys were never dropped in. I could pretend that was discipline. It was not, it was a config gap, and I would rather say so than construct a story about patience. I wrote up the full promotion sequence in the paper to live trading guide, and the journal I log trades in is a free offline R-multiple app with no account and no backend.

How to run this test on your own system

This is the part worth stealing. It takes an afternoon and it is the highest value afternoon you will spend on a strategy.

1. Express results in R, not dollars or percent. Otherwise position sizing quietly contaminates the performance number and you cannot separate the two effects.
2. Build the random arm properly. Same universe, same bar frequency, same stop distance distribution, same exit rules, same trade count, same date range. If your random arm risks differently than your real arm, you have benchmarked nothing.
3. Run it many times. One random seed is an anecdote. Run enough repetitions to get a distribution, then compare against that distribution rather than against a single run.
4. Report the difference, with an interval. Strategy expectancy minus benchmark expectancy, with a confidence interval on the difference. That number, not the headline expectancy, is your actual claim.
5. Decide the gate before you see the answer. Write down what result would make you trade it and what result would make you stop, while you still do not know which one you are about to get.

Why publish a failure

Because the absence of negative results is exactly why retail traders keep rediscovering the same dead ideas. If every writeup you read is a winner, you conclude that finding winners is easy, and then you interpret your own losses as a personal failing rather than as the base rate.

There is also a selfish reason. I am a computer science student applying to quant and software internships, and I would rather be the candidate who ran the placebo arm on his own work than the candidate with a prettier number and no benchmark. One of those survives a follow-up question.

The full walk-forward writeup, including what happened to VCP and the filters, is in the earlier post on what walk-forward did to my strategies. The screener itself, the position sizer, and the free risk tools are on the risk tools page.