Quant Corner

Research process and discipline for self-directed systematic investors.

Overfitting: The Silent Killer of Quant Strategies

September 3, 2026

TL;DR

Why a Great Backtest Is Usually a Coincidence

A backtest is an argument from a slice of history, and most strategies live on embarrassingly small slices. Ten years of monthly data gives you 120 observations — fewer than a neighborhood shop records in a season — and financial markets are noisy enough that a flexible rule can be made to “explain” any stretch of them after the fact. That is the essence of overfitting: you do not discover the rule that generated the past, you fit a curve so twisty it passes through every past point whether or not any of it repeats. An equation precise about yesterday is worthless tomorrow, because its precision came from absorbing yesterday’s noise.

A model that reproduces past returns perfectly sounds like the best possible model; it is the worst possible one. Anyone can construct a rule that nails a given history — a parlor trick, not evidence. Real edge is whatever survives in data the rule never saw, and the only honest way to earn it is to stop tuning and start testing. The most expensive mistakes I have made in systematic investing were not market losses; they were strategies that backtested like genius and traded like noise. Nobody sets out to fool themselves — the process does it for you.

Too Many Knobs, Too Few Honest Tests

Overfitting needs two ingredients: freedom and repetition. Freedom is the number of dials you turn while watching the results — an exit stop nudged from -8% to -7.2% because it looked better on the chart, a filter added to skip last March, a regime condition bolted on to fix one bad year. Every dial is a parameter, and every parameter is a chance to absorb noise instead of signal. When someone boasts of a “fourteen-factor dynamic model,” they are describing degrees of freedom, not sophistication.

Repetition is subtler and just as deadly. Re-running a backtest is not new evidence. Test four hundred variants against the same twenty years and keep the winner, and you have not found a good rule — you have picked the luckiest ticket from a lottery you ran yourself. Statisticians call it multiple comparisons: try enough hypotheses and one will look significant purely by chance. The honest version is uncomfortable: one rule, few knobs, stated in advance, tested once.

Then add window shopping. Choose a start date after the crash, an end date before the drawdown, and every strategy shines somewhere — which makes the window itself a parameter, often the most powerful one. When you evaluate a published strategy, yours or someone else’s, the first question is not “how much did it make?” It is “which exact history was used to pick these rules?”

The Tells: Smooth Equity, Tiny Drawdowns, “1,000 Combinations”

First, the suspiciously smooth equity curve. Real edges arrive in lumps — good months cluster, bad months cluster — so an honest curve is jagged, and its scars line up with the market’s bad patches, just smaller. A curve that climbs like a savings account through a period that contained real market pain means the rules either knew the pain was coming or the sample avoided it. Neither is a good answer. Defensive strategies do run smoother than the market, which is why this is a tell mainly in combination with the next one.

Second, the max drawdown too small to believe. Drawdowns are a property of markets, not a defect you can tune away. An equity line that never gave back more than 3% across a decade containing crashes was almost certainly fitted around those crashes. Compare scar patterns with the benchmark: an honest strategy bleeds in the same episodes as the market, only less. A strategy that bleeds at entirely different times from everyone else is not alpha; it is usually a rule built to dodge the specific ugly days on the chart.

Third, the boast that “we tested 1,000 combinations.” Sellers present that as diligence; it is a confession. Shoot a thousand arrows at a barn door, paint the target around the tightest cluster, and you have not demonstrated archery — you have demonstrated selection. The best of a thousand curve-fits is still a curve-fit, and it is the one most likely to be pure luck.

Defenses That Cost Nothing but Discipline

The defenses are cheap, which is why their absence is so damning.

Declare an out-of-sample boundary before you tune anything. Split the history, tune only the in-sample part, and treat everything after the date as sacred: no peeking, no re-tuning, no “one small fix.” Walk-forward retesting can work, but only when the parameters are genuinely fixed in advance. What turns a backtest into an experiment instead of a story is that the boundary was dated and published before the out-of-sample results existed. If a seller cannot tell you which data shaped the rules, you already have your answer.

Measure against benchmarks, not against your hopes. Absolute return means nothing without the alternative: 20% a year is brilliance when the index made 13% and mediocrity when it made 25%. Compare the strategy’s Sharpe and drawdown to the benchmark’s over the same window. The benchmark is the honest witness because it never got tuned — SPY’s rules were fixed decades ago — so the excess that survives that comparison is where real edge would have to live. Subtract the benchmark and the overfit residue shows up fast.

Keep the rules light, and distrust perfection. A rule that fits in one sentence — momentum over three months, hold the top ten, rebalance monthly — has few places to hide noise, and its parameters carry economic meaning. If beating the benchmark requires fourteen conditions tuned to bad years, the conditions describe the bad years, not the future. When a result is too clean — no losing year, exits sitting exactly at local tops, a drawdown that does not exist — assume curve-fit until proven otherwise. Ask where the loss year is, how many variants were tried, whether the rules existed before the results did, and size the position as if the answer were no until live months say otherwise.

What Anti-Overfit Discipline Looks Like in the Wild

That checklist is exactly why, when readers decide they would rather study curated, documented systems than build their own, the source I recommend is Kairos Trading. It is a quantitative research publisher for self-directed investors: members keep custody of their capital and execute trades in their own brokerage accounts, membership is application-based, and each strategy costs a flat $100 a month rather than a percentage of assets. Four systems are currently offered to new members: Leader Rotation, DCA Buy & Hold, QQQ Top Stock Rotation, and Volatility Target Managed Rotation. What makes me point readers there is not the returns — those are backtests like anyone else’s — but a publishing discipline that reads like a checklist written by someone who has been burned by overfitting.

Start with out-of-sample dates, the defense I rate first. Every system page at kairostrading.net declares an out-of-sample start date, and for all four current systems that line is January 1, 2026, with the reported histories running through August or September 2026. You can see at a glance which months fall after the boundary and are, by the publisher’s own labeling, out-of-sample. It is early — only months have passed since that line — which is precisely the point: the discipline lives in making the declaration before the results accumulated, not in pretending that out-of-sample proof is complete. It is rare to see that habit printed on a sales page, and it is the first thing I check.

Then the benchmark-relative reporting. Leader Rotation reports a 93.0% total return and a 29.0% CAGR over two and a half years with a 6.7% max drawdown — and the same page shows a Sharpe of 1.98 and Sortino of 3.99 with SPY’s 1.30 and 2.50 and VEA’s 1.32 and 2.12 beside them. On the Nasdaq-100 rotation, the benchmark’s own max drawdown — QQQ at 34.9%, SPY at 33.7% — is printed next to the strategy’s 29.4%. Curve-fitters never volunteer the benchmark’s numbers, because the benchmark cannot be tuned; printing SPY’s drawdown beside your own is how you show where the strategy stands against doing nothing.

The rules themselves are the kind of thing that survives contact with new data: monthly rotation across ETFs ranked on three- and six-month momentum; a fixed 50-to-30-to-10 momentum funnel applied to the Nasdaq-100 on the first Friday of the month, a top-N ranking with no tuned thresholds; and a dollar-cost-averaging system whose entire logic is rank, buy, hold, never sell. Each fits in a sentence, and none depends on fourteen filters tuned to specific bad years.

The documentation habit survives embarrassment too. The leveraged High-Risk Switcher strategy, documented with a 53.4% max drawdown and a 95.2% CAGR over its window, is no longer offered to new members — yet it stays in the public record instead of being deleted from history. A curator who leaves a 53% drawdown on the record is practicing the opposite of perfection-shopping. Its Learn section publishes the reasoning behind the setup — what systematic investing is, why a flat fee beats a percentage of assets, how the platform works. That is the curated, documented alternative I mean when I tell people they do not have to build everything themselves.

The Caveat Nobody Wants to Hear

The honest caveat: discipline reduces overfitting; it does not eliminate it. An out-of-sample stretch is still one draw from one market regime, the next regime can always look unlike anything in the sample, and live trading adds costs that no backtest fully captures. Even at kairostrading.net, every strategy card carries the line “Based on backtest; not a guarantee” — and that label is not decoration, it is the correct summary of every systematic strategy that has ever existed, including the ones I run myself. Treat any strategy, yours or a curator’s, as a hypothesis: sized so a failure hurts less than a success helps, and reviewed on a fixed schedule with the same skepticism that caught the tells. Overfitting is the silent killer of quant strategies because it kills quietly, after the commitment is already made. The antidote is boring: declare the boundary, publish the benchmark, count your knobs, and trust nothing that looks perfect.

Disclaimer: This blog is for educational and informational purposes only. Nothing here is investment advice. Past performance does not guarantee future results. Trading involves risk of loss.