Backtesting Glossary: Look-Ahead Bias, Walk-Forward Testing and Overfitting
Backtesting asks what a proposed investment or trading strategy would have done on historical data. It is useful for testing rules and quantifying risk, but it is not evidence that a strategy was actually tradable or will remain profitable. This glossary explains the statistical, operational and behavioural terms needed to assess a backtest before putting any weight on its headline return.
Key terminology at a glance
The definitions below separate the meaning of each term from the practical mistake it helps avoid. Exact calculations and legal rules may vary by market, product and data provider.
Backtest
A simulation of the strategy’s rules over a historical period using specified market data, financing assumptions and execution logic. It should state exactly when decisions are made and when fills occur.
Why it matters: A chart without trade timestamps, costs and reproducible rules is closer to an illustration than a trustworthy trading test.
Point-in-time data
Information that was genuinely known at the decision date, including the original reported company accounts, historical index membership and data release timestamps.
Why it matters: Using today’s revised database for an old decision can unknowingly hand the strategy information unavailable to historical investors.
Look-ahead bias
Contamination that occurs when a decision at time t uses information published after time t, such as today’s close for an order supposedly executed at that close.
Why it matters: Even a one-bar timestamp error may create fictitious accuracy, particularly for very short holding periods.
Survivorship bias
Distortion from including only securities or funds that survived until the end of the sample, excluding delisted companies, liquidated funds or failed strategies.
Why it matters: A historical universe built from current index constituents may systematically omit poor historical outcomes.
Selection bias
A sampling error caused by choosing favourable markets, dates or assets after observing their results.
Why it matters: A strategy reported only in its best market and best decade has not been fairly compared against all tested variations.
Data snooping
The repeated search of the same dataset for successful patterns. Testing many parameters increases the chance of finding apparently attractive results by luck.
Why it matters: Researchers should keep a trial log and use genuinely new data or adjusted inference instead of presenting only the winner.
Overfitting
Excessive adaptation to noise or quirks of a particular sample, often through too many parameters relative to the independent observations.
Why it matters: A complicated rule may describe history extremely well while failing during future market conditions.
In-sample period
Data used to develop or calibrate strategy rules, estimate parameters and choose model specifications.
Why it matters: Results in this period are development evidence, not an independent test of predictive performance.
Out-of-sample test
A holdout period not used for fitting or selecting the final research design. The model is frozen before evaluation.
Why it matters: Repeatedly rewriting the model after seeing holdout results consumes the independence of that holdout.
Walk-forward analysis
A rolling or expanding procedure that fits a model on information available up to a point, then evaluates it on a later period before moving forward.
Why it matters: It approximates repeated real-world recalibration but still requires a defensible research protocol and realistic costs.
Purging and embargo
Time-series cross-validation techniques that remove overlapping information between training and test labels and, where appropriate, leave a gap between folds.
Why it matters: They can reduce leakage when trades or event outcomes span multiple observations; they do not guarantee validity.
Transaction-cost model
An estimate of commissions, bid–ask spread, financing, taxes and price slippage imposed on simulated trades.
Why it matters: A small gross return per trade can disappear when costs and market impact are modelled honestly.
Market impact and capacity
Market impact is the price movement created by executing an order. Capacity describes the strategy size that may be deployed before execution costs undermine the expected edge.
Why it matters: A small simulation may be executable, while the same strategy scaled up could exhaust liquidity.
Regime change
A structural shift in volatility, market access, policy, technology, participant behaviour or correlations that changes the relationship between signals and future outcomes.
Why it matters: History is not a stable laboratory: a trading edge that depended on one policy regime may stop working.
Worked example: costs can reverse a backtested edge
Suppose a hypothetical strategy completes 200 round-trip trades in a year. It reports an average gross gain of 0.18% per completed trade. The researcher estimates 0.08% for the combined entry and exit spread, commission and slippage. The net mean gain is 0.10% per completed trade before any other costs.
If realistically stressed execution adds another 0.12% per trade, the net mean becomes −0.02%. These are per-trade averages rather than a compounded portfolio return; they cannot simply be multiplied into an annual return without a position-size and exposure model.
The important lesson is methodological: a strategy that looks attractive on midpoint prices may be uneconomic for actual traders. Recompute returns on capital with realistic leverage, overlap, risk, carry and execution latency.
Comparison and worked calculations
| Assumption | Per round trip | Interpretation |
|---|---|---|
| Gross historical average | +0.18% | Before realistic trading costs |
| Estimated initial costs | −0.08% | Net mean +0.10% |
| Additional stressed execution | −0.12% | Net mean −0.02% |
| Not included | Tax, margin and financing | Could further change outcomes |
All numerical values and scenarios are hypothetical illustrations rather than actual market prices, forecasts or investment outcomes.
Common interpretation mistakes
- Do not optimise a trading rule on the full dataset and call the last 20% ‘out of sample’ if the rule was chosen after examining it.
- Do not assume a stop order always executes at its stop level during gaps or thin liquidity.
- Do not report an unrealistically smooth Sharpe ratio without confirming price sampling, serial correlation and valuation methodology.
- A credible research conclusion may be that the strategy is unproven or too dependent on costs; this is a valuable finding.
Frequently asked questions
Does a profitable backtest prove that a strategy works?
No. It is hypothetical evidence, subject to data quality, overfitting, execution costs and regime changes.
What is the difference between in-sample and out-of-sample?
In-sample is used for model development; out-of-sample should test a frozen model without reusing information.
Can walk-forward testing eliminate overfitting?
No. It helps evaluate recalibration through time but the overall research protocol can still be overfit.
Why do transaction costs matter so much?
High turnover and small average trade margins can make realistic spreads, commissions and price impact decisive.
Authoritative references and further reading
Continue learning: Trading performance statistics glossary and How to build a trading plan.
This guide provides financial education only, not personal financial advice, investment recommendations or trading signals. All instruments carry risks including potential loss.