Funds & Trackers
Backtests describe a past that was chosen after the fact
Any strategy can be made to look excellent in history if the rules are selected once the history is known.
This is less a set of instructions about backtested strategies than an argument, and it is worth saying so at the start.
The argument in brief
- Rules fitted to a known history will flatter themselves on that history.
- Only the surviving strategies get published.
- A rule with no reason behind it is a pattern, not a mechanism.
How a good backtest is produced
Given a fixed history and freedom to choose the rules, thresholds and start date, it is straightforward to produce an impressive-looking record. Nothing dishonest is required; testing many variants and presenting the best is enough. The result describes how well the rule fits that history, not how it will behave on data it has not seen.
This is why the second question about any backtest is how many alternatives were tried.
Survivorship in the results you see
Strategies and funds that performed poorly are quietly discontinued and stop appearing in the data. What remains is a record of the survivors, which makes the category look better than the full population was. The same applies to published research, where results showing nothing are less likely to appear.
In practice, any set of impressive historical records should be read with the question of what is missing from it.
Mechanism separates a rule from a coincidence
A strategy worth taking seriously has an explanation for why it should work that does not depend on the data it was found in. Explanations usually involve someone accepting a risk others avoid, or a persistent behaviour by other investors.
Where the only argument is that it worked in the past, the pattern and the coincidence are indistinguishable. Even with a mechanism, effects can shrink once they are widely known and acted upon.
The behavioural finance caveat applies here too
Several well-known market anomalies have weakened or disappeared in data after their publication. Some findings in the wider behavioural literature have also replicated less reliably than their popular reputation suggests. This does not discredit the field, but it does argue for treating any single striking result as provisional.
Honest uncertainty about magnitudes is more useful than a confident number that turns out to be fragile.
What it means for you
A product marketed on historical simulation is offering the weakest form of evidence, presented in the most persuasive format. Charts are unusually convincing because they show a smooth line where the reality was a sequence of decisions nobody actually made. Asking whether you would have held it through its worst period is a better test than looking at the endpoint.
On an ordinary week, the worst period is normally in the small print rather than the headline.
Some of this will suit you and some will not, and that is the point.
Using history properly
History is genuinely useful for understanding the range of outcomes: how deep falls have been, how long recoveries have taken, how often bad years occur. That use does not require fitting rules to it and is not vulnerable to the same problem.
Planning for a repeat of the range is reasonable; planning for a repeat of the sequence is not. The distinction is between calibrating expectations and predicting a path.
The takeaway
Ask what the rule would have to be wrong about. If nothing, it was fitted to the past.
Small and repeatable beats ambitious and abandoned, almost every time.
Questions readers ask
Is all historical analysis useless?
No. Using history to understand how bad things have been and how long they lasted is sound. Using it to select rules that would have worked is where the trouble starts.
How can I judge a backtested claim?
Ask for the mechanism, the worst historical period, and how many variations were tested. The third question is the one most rarely answered.
