Glasshouse Research · June 2026 · 6 min read
Backtest vs live: what ~5,000 simulated trades taught us
Across 21 strategies and roughly 5,000 simulated trades — some tested on up to 12 years of market data — we've learned one lesson more reliably than any other: the backtest is the audition, not the performance. Live markets routinely disagree with simulations, and the gap is where most trading services quietly lose their customers' money.
Because our whole model is built on showing the gap rather than hiding it, here's what it actually looks like in our own book.
The gap, in our own numbers
From our live strategy book (full table
here):
· A smart-money-concepts system: backtest profit factor
1.22 over 2,299 trades → live
0.46 over 21 trades.
· A money-flow long system: backtest
1.19 → live
0.11 over its first 10 trades.
· Our flagship higher-timeframe momentum system: backtest
1.31 → live
1.67 over its first 7 — early-positive, sample still small.
Two strategies materially worse live; one better so far. That spread isn't bad luck — it's the normal physics of the gap, and any service showing you only its backtests is hiding exactly this.
Why backtests flatter
- Perfect fills don't exist. Simulations assume you traded the price on the chart. Live, there's slippage, spread, partial fills and fees — and they always point the same direction: against you. Thin edges die here first.
- Overfitting is the default, not the exception. Test enough parameter combinations and some will look brilliant on history by pure chance. The market never saw your parameters; history did.
- Regimes change. A strategy tuned on a trending market meets a choppy one. The rules didn't break — the world they were fitted to ended.
- Small live samples mislead in both directions. Seven good trades don't prove an edge; ten bad ones don't always kill one. The discipline is deciding in advance what sample size justifies what conclusion.
How we gate capital because of this
Every Glasshouse strategy climbs a ladder, and the gap is exactly why the ladder exists:
- Research — walk-forward backtests on years of data; thin edges (profit factor near 1.0) are rejected outright.
- Paper — the strategy trades live markets with fake money. This is where simulation meets real order flow, and where most candidates die.
- Testnet / small live — real execution, capped risk, pre-committed kill criteria watching every trade.
- Live capital — only strategies that survived all of the above, and they keep their kill criteria forever.
A backtest gets a strategy an interview. Only live trading gives it the job — and it can be fired at any time.
What this means if you're evaluating any trading service
Ask one question: "show me backtest and live, side by side." If they show only backtests, they're auditioning. If they show only a hand-picked live streak, they're cherry-picking. The honest answer is both columns, for the whole book, losses included — which is precisely what our research page is.