Five backtesting engines, same strategy, different results.
Yin, Miki, Lesnichenko, and Gural (arXiv:2603.20319) formalize “implementation risk” — the divergence in backtested portfolio metrics that arises solely from how engines implement the same logical strategy. They test 15 benchmark strategies across five open-source engines on 180 S&P 500 stocks.
At zero transaction costs, all five engines agree perfectly. Under nonzero costs, divergence reaches up to 3.71 percentage points for high-turnover rotation strategies. The source of disagreement is not the strategy definition — every engine receives the same specification — but the execution details: how fills are modeled, when prices are sampled, how dividends are handled, when costs are applied. These implementation choices are typically undocumented and vary between engines.
The metrological framework is precise: engine sensitivity (how much metrics change per engine swap), implementation uncertainty interval (the range of plausible results), divergence amplification factor (how implementation differences scale with strategy complexity), and conclusion stability index (whether engines agree on the direction of a metric). For the tested strategies, the conclusion stability index equals 1 — engines disagree on magnitude but agree on sign. The strategy that backtests as better on one engine backtests as better on all of them.
This means the implementation gap is a noise problem, not a bias problem — at least for the strategies tested. But the 3.71% maximum divergence is larger than many strategy improvements claimed in quantitative finance. A researcher who reports a 2% annualized improvement using one engine might find the improvement vanishes on another. The improvement is real only if it exceeds the implementation uncertainty interval.
Forensic analysis uncovered seven undocumented defects across three engines, grouped into five failure modes. The engines have bugs that affect backtested results in ways their users don't know about. The tool is part of the measurement.