In-sample vs out-of-sample: what is the difference?
In-sample results come from the same data used to select the model's rules and parameters. The model has, in effect, already seen the answers. Out-of-sample results come from data held back entirely, so the model is being asked a question it has not been shown.
Both are worth reporting, but only one of them is evidence. Presenting in-sample results without labelling them as such is one of the most common failures in quantitative market research.
What is walk-forward testing?
Walk-forward testing is the stricter version. Instead of holding back one block of data, the model is repeatedly fitted on a period and evaluated on the period immediately after it, rolling forward through history.
The point is to reproduce the constraint that real use imposes: at every historical decision point, the model may only use information that existed at that moment. It cannot borrow anything from the future, including the knowledge of which conditions later turned out to matter.
Why overfitting is so easy
Given enough parameters and enough attempts, any historical series can be fitted almost exactly. The result describes the past in detail and generalises to nothing.
- Too many parameters relative to the number of genuine events in the history.
- Repeated re-testing until a configuration looks good — the test set quietly becomes a training set.
- Look-ahead bias — using data or revisions that were not available at the time.
- Survivorship bias — testing on a universe that excludes what failed.
- Single headline numbers that hide how the model behaved in each distinct regime.
What honest validation reporting looks like
Credible validation labels in-sample and out-of-sample results separately, describes the testing procedure well enough to be reproduced, and reports behaviour per regime rather than as one blended figure.
RegimeSignal™ publishes its methodology and independent validation for that reason, and separates in-sample from prospective results wherever they are shown.
Frequently asked
What is an out-of-sample backtest?
A test of a model on data that was not used to build or tune it, so the result reflects how the model handles conditions it has not already been fitted to.
What is the difference between in-sample and out-of-sample results?
In-sample results come from the data the model was fitted on and are usually flattering. Out-of-sample results come from held-back data and are the meaningful test of whether a model generalises.
What is walk-forward analysis?
A rolling form of out-of-sample testing in which a model is fitted on one period and evaluated on the next, repeatedly through history, so every call is judged using only information available at that time.
Why do backtests often fail in live use?
Usually because of overfitting, look-ahead bias or repeated re-testing that turns the held-back data into training data. Each makes historical results look better than the model's real ability.