What Your Backtest Isn't Telling You: A Practitioner's Checklist
How to spot hidden pitfalls in quantitative strategy results — before you risk real capital.
Key Takeaways
01Backtest results are only as reliable as their design — common pitfalls like survivorship bias and look-ahead bias can dramatically overstate performance if left unchecked.
02Overfitting and multiple testing inflate the risk of 'discovering' strategies that work only in-sample; robust out-of-sample validation is essential.
03Assumptions about transaction costs, liquidity, and slippage often make or break a strategy's real-world viability — always scrutinize these inputs.
04A systematic checklist can help investors critically evaluate any backtest, whether it's their own or presented by others.
If you've ever been dazzled by a backtest showing eye-popping returns, you're not alone. But as any seasoned quant will tell you, the real world is far less forgiving than a spreadsheet. The difference between a robust strategy and a statistical mirage often comes down to the details most backtests leave out — or gloss over.
Before you trust any backtest (including those on AIBROKER), it's worth running through a practitioner's checklist. Here are the questions that separate credible research from wishful thinking.
1. Survivorship Bias: Are Dead Stocks Missing from the Sample?
Survivorship bias occurs when a backtest includes only assets that survived to the end of the sample period, ignoring those that went bankrupt, merged, or delisted. This can inflate historical returns, especially in equity strategies.[1]
For example, a momentum strategy run on today's S&P 500 constituents will look much better than one that includes all stocks that were ever in the index, including failures. Always ask: Does the backtest use a survivorship-free universe? If not, results may be overstated. For a deep dive, see our article on survivorship bias.
2. Look-Ahead Bias: Is Future Information Sneaking In?
Look-ahead bias happens when a backtest inadvertently uses information that would not have been available at the time of trading. This can be as subtle as using annual financials before their release date, or as blatant as rebalancing on data that wasn't published yet.[2]
To avoid this, ensure all signals and universe definitions are lagged appropriately. If a strategy claims to use "real-time" data, check the data vendor's documentation for reporting lags and revisions.
3. Overfitting and Multiple Testing: Is the Strategy Too Good to Be True?
Overfitting occurs when a model is excessively tuned to historical data, capturing noise rather than signal. The more parameters or variations you test, the higher the risk of finding spurious results. Harvey, Liu, and Zhu (2016) estimate that most published finance factors are likely false positives due to multiple testing.[3]
A robust backtest should limit the number of parameters, report all variations tested (not just the best), and use out-of-sample or walk-forward validation. See our momentum premium article for an example of transparent versus opaque backtest reporting.
4. Transaction Costs, Slippage, and Liquidity
Ignoring transaction costs and slippage can turn a seemingly profitable strategy into a money-loser in practice. Always check: What assumptions were made about commissions, bid-ask spreads, and market impact? Are these assumptions realistic for the strategy's turnover and asset class?
Impact of Transaction Costs on Sharpe Ratio
Transaction Cost (bps)
Gross Sharpe
Net Sharpe
0
1.20
1.20
10
1.20
1.00
25
1.20
0.80
50
1.20
0.50
Illustrative data for a hypothetical monthly-rebalanced US equity momentum strategy, 2000–2020. Assumes 500 stocks, equal-weighted, no slippage, risk-free rate 2%. Not actual performance.
5. Out-of-Sample Testing: Has the Strategy Been Validated on Unseen Data?
The gold standard for any backtest is out-of-sample validation — testing the strategy on data not used in model development. This can include walk-forward analysis, holdout periods, or live paper trading.
If a backtest only reports in-sample results, skepticism is warranted. For more on robust performance metrics, see Sharpe vs Calmar: Which Ratio Matters?
A Practitioner's Backtest Evaluation Checklist
In markets, skepticism is a virtue. The best investors aren't just data-driven — they're data-skeptical. Next time you see a backtest with dazzling returns, remember: what matters most is not what the backtest shows, but what it might be hiding.