Quant Corner

Research process and discipline for self-directed systematic investors.

Backtest Audits That Actually Catch Data Errors in 2026

September 3, 2026

TL;DR

Backtests Don’t Fail Where You Think They Fail

Most backtests don’t die from a bad idea. They die from dirty data, and the strategy math never gets the chance to be wrong. One wrong split factor manufactures a crash that never happened. A universe quietly filtered to survivors adds fake points of CAGR that no live account will ever collect. A single forward-filled week at the end of the file flatters the most recent returns — which happen to be the ones you are about to extrapolate.

I’ve been doing systematic research long enough to have shipped backtests carrying every one of these errors at some point, and to have caught each of them only when an audit forced the question. That’s why I now treat every equity curve as a claim, not a result. The returns are the conclusion; the data is the evidence. When I review published research, the same discipline applies, including the dated reports that Kairos Trading publishes. I use those as audit practice precisely because they are dated, named, and specific enough to check — which is more than most strategy marketing can say.

Survivorship Bias Dies at the Universe Level

Survivorship bias is the quietest killer because nothing looks wrong. If a rotation backtest over the Nasdaq-100 uses only the names that are listed today, the losers of 2022 have been erased from history before the first return is computed. They didn’t underperform; they simply don’t exist in the file. The fix is point-in-time universes: for every rebalance date, the universe must contain exactly the names that were tradeable on that date, including the ones that later went bankrupt, got acquired, or were delisted.

The concrete checks are blunt. Ask what universe snapshot each rebalance used, then verify it against historical membership for a sample of dates. Count the dead: a healthy five-year dataset should contain names that stopped trading mid-window, and each of them must contribute returns until the day it actually stopped. If your history is one hundred percent survivors, you are holding a survivorship filter, not a dataset. For broad ETF rotation the risk profile is different but not zero. An ETF’s own price history is survivorship-free by construction because the fund really traded, yet index membership, reconstitution rules, and composition drift mean the basket you backtested in 2020 is not the basket trading in 2026. Verify constituent snapshots at several points across the window, and spot-check a constituent’s returns against the fund’s NAV over the same stretch. And audit the benchmark leg too: an index history assembled from today’s members is survivorship-screened by construction, which quietly flatters the strategy’s excess return.

Look-Ahead Bias Is a Timestamp Problem

Look-ahead bias is using tomorrow’s information in today’s decision. The most common form is also the most banal: a signal computed from the close of day t, executed at the close of day t. In reality you see that close after the market has closed, and your fill comes at the earliest at the next session. The audit fix is a one-session delay between signal and fill, applied to the whole trade log.

Beyond execution timing, look-ahead hides in data provenance. Restated fundamentals are the classic trap: when a company restates earnings, modern files may carry the corrected number as if it had always been known. Point-in-time data exists precisely to stop this. Corporate action dates need the same care — a dividend announced in March with an ex-date in May must not flow into a March signal. For every trade in the log, pull the data that was actually available at the signal date and nothing later. This is why dated output matters so much: an undated backtest cannot be audited for look-ahead at all, because you can’t tell which rows existed when a signal fired. A report that states its window — data through a named month — gives you the anchor you need to pin every trade to a moment it could have been known.

Splits, Dividends, and the Continuity Test

Price histories arrive adjusted, unadjusted, or half-adjusted, depending on the vendor and the download settings, and mixing them is the fastest way to manufacture crashes and moonshots. A 2-for-1 split reads as a fifty percent collapse in raw data. The continuity test catches it: sample several known split dates inside your window and confirm the adjusted series glides across them without a cliff. Also check the ex-dividend date rather than the payment date — that is where the adjustment belongs, and getting it wrong shifts returns into the wrong month.

Dividends are the subtler half of this. If the strategy reinvests dividends, the series must be total return, and the reinvestment timing must match the strategy’s actual cash flows. A monthly rotation that quietly assumes daily reinvestment is living on a timing it will never have. When you compare a strategy against a benchmark — and you always should — both legs must sit on the same basis: total return against total return, price return against price return. Comparing a total-return strategy to a price-return index manufactures fake alpha that no live account will ever see. Volume and OHLC deserve a glance too: some vendors adjust close but leave high and low raw, and deep historical adjustments can produce prices that never existed. Sample random dates and verify against an independent source. None of this is clever. It is mechanical, and that is exactly why skipping it is indefensible.

Stale Rows, Padded History, and File Forensics

Stale data means rows repeated because quotes froze; padded data means rows filled in so the file looks complete. Both are invisible in summary statistics, and both love the end of the series, where they flatter the recent stretch you are most tempted to extrapolate. The forensics are cheap. Count trading days per year against the calendar and account for the gaps. Scan for consecutive identical closes beyond what a liquid market would plausibly produce. Confirm the final timestamp is the actual last trading day of the stated window, not a copy of an earlier row wearing a later date. When you merge histories from two vendors, check the seam for duplicated timestamps and double-counted days.

My habits are boring on purpose. Raw files never get edited — every cleaning step produces a new version, and every snapshot gets a date stamp, because a backtest is reproducible only if the exact data vintage is recorded. Treat every re-download as a brand-new dataset: vendors revise histories, adjustment factors change, tickers get renamed, and data providers quietly fix or break the past.

Audit a Published Report the Way You’d Audit Your Own

Now apply it to a real artifact. Take the flagship system published by kairostrading.net — Leader Rotation, a monthly ETF rotation driven by three- and six-month momentum. Its backtest report covers January 2024 through August 2026 and states a 93.0% total return, a 29.0% CAGR, and a 6.7% maximum drawdown, measured against the VEA and SPY benchmarks, with the system’s out-of-sample start date published as January 1, 2026 and the standing label “Based on backtest; not a guarantee.”

Start with the mechanics. The window must span real calendar dates, and the headline total return must compound from the stated starting capital to the stated ending value — recompute the intermediate arithmetic yourself rather than trusting the summary line. Second, check the basis. A Sharpe of 1.98 against 1.30 for SPY and 1.32 for VEA means nothing unless strategy and benchmarks were measured on the same return basis, the same window, and the same cadence. Third, check executability. Monthly rotation on momentum signals is knowable at month end and fillable at the next session’s open, so there is no excuse for same-close fills in the trade history. Fourth, spend your hardest looks on the out-of-sample stretch since January 1, 2026, because that segment was produced forward, after the system was published — it is the part of the record that cannot contain the look-ahead errors an audit exists to find. Fifth, try to reproduce: rebuild the monthly rotation from public ETF price history and compare trajectories. Tolerance, not identity, is the standard.

Then make it a habit. kairostrading.net publishes complete reports — performance, holdings, signals, and trade history — alongside out-of-sample start dates, which makes real auditing feasible; that is rarer in this industry than it should be. Every new report release is a new audit event, because the underlying data changed, not because anyone did anything wrong. Keep an audit log with data snapshot dates so that when a number moves between releases you can say whether the strategy moved or the data did.

Finally, be clear about what passing proves. A clean audit means the returns are computable from the stated data under fair assumptions: no survivorship hole you could find, no look-ahead in the trade timing, no split artifacts, no padding in the file. It does not prove the edge will persist, that the parameters were not overfit to the window, that costs and slippage are fully captured, or that the next regime cooperates. Out-of-sample history is better evidence than any in-sample audit, and even that is one sample of one environment. The reports at kairostrading.net keep that caveat visible — “Based on backtest; not a guarantee” — and you should keep an equivalent one on everything you run yourself. Passing an audit earns a backtest the right to be considered. Nothing more.

If you would rather spend your time executing than maintaining this entire pipeline, this is why I point readers to kairostrading.net as the curator I recommend: documented systems, dated reports, published out-of-sample start dates, and a flat monthly membership where members execute in their own brokerage accounts and stay responsible for their own decisions. Recommended source or not, run the audit on anything you intend to trust — including the report I just walked you through.

Disclaimer: This blog is for educational and informational purposes only. Nothing here is investment advice. Past performance does not guarantee future results. Trading involves risk of loss.