Distribution shift: performance gets worse over time
A model that worked and then stopped is usually the world moving, not the code breaking. The four things that drift, the diagnostics that tell them apart, what each does to the score and to the probabilities, and the related trap where the aggregate relationship differs from the one inside every group.
“The model got worse” has four different meanings, and the fix for each is different. Before touching the model, find out which one it is; each has a diagnostic that takes one line.
The four kinds of drift
| What moved | Diagnostic | What it does | Example |
|---|---|---|---|
| the base rate (the target’s mean or positive share) | target mean by month or year | probabilities come out too high or too low everywhere; the ranking may survive | bank marketing: 5% to 17% to 52%; flight delays: summer 30%, autumn 14% |
| the level of a series (trend, growth) | monthly mean of the target | a model without level-carrying features is wrong in one direction for the whole period; a shuffled split hides it | bike rentals: rentals doubled in two years |
| the population (who is in the rows) | feature distributions by period: category shares, numeric quantiles | the model is asked about rows unlike the ones it learned from | bank marketing: 46% “unknown” contact method in 2008, 0% in 2009; previous-campaign outcomes known for 3% then 66% |
| the relationship (the same features now mean something else) | the model’s error, or the sign of a key coefficient, by period; refit on each period and compare | the ranking itself degrades; no re-anchoring fixes it | reversal by year: an effect present in some years and absent in others |
Run the first three diagnostics before fitting anything. They are the yearly-means table, the monthly plot, and a crosstab of the main categoricals by period. Together they take a minute and they decide whether the study is one problem or several.
What each does to the numbers
- Ranking metrics (AUC, rank correlation) survive a base-rate shift and a level shift, because they only compare rows within the test period. They degrade under population shift when the new rows fall where the model has little training data, and under relationship shift always.
- Calibration is the first casualty of any of them. The flight-delay model ranks autumn departures as well as summer ones and over-predicts every decile; the hotel model under-predicts 2017 because 2017 cancelled more. A model whose probabilities will be used needs its base rate re-anchored on recent data, and a plan to do it again.
- Calendar columns can turn a base-rate shift into a relationship shift. A month column trained on two years learns “May is bad” from one enormous May campaign and applies it to a different May; dropping it improved the chronological AUC in the bank data. If a month effect does not repeat across years, it is memorising, not learning.
The test that shows it
Walk forward. Fit on the data up to each period boundary, score the next period, and tabulate the score by period against the baseline’s score by period. Three outcomes:
- Model and baseline both degrade together: the problem got harder (the series became noisier, the population changed). The model’s margin over the baseline is what to watch.
- The model degrades and the baseline does not: the model learned something that stopped being true. Look at which features did the work in the early periods.
- Neither degrades: no drift; the chronological score is the score. The traffic study is this case, and its margin over the seasonal naive grows with the training window.
The load-forecasting study tabulates nine months this way; the reversal study does it by year with an IC.
Grouped confounding
A related trap that does not involve time. The relationship in the whole table can differ from the relationship inside every group, when group membership is correlated with both the feature and the target.
The house-price study has a clean instance: houses with the lot frontage missing sell for 7% more than houses with it recorded, and within every neighbourhood the difference is zero. The missingness clusters on cul-de-sac lots, which cluster in expensive neighbourhoods. The aggregate correlation is real and it is about neighbourhoods, not about frontage. The same shape appears whenever an aggregate association is claimed on a table with an obvious grouping column: check the within-group version before believing the between-group one, and say which one the claim is about.
The tools are the ones from the panel studies: compute the relationship within groups (demean by group, or one regression per group and look at the distribution of slopes) and between groups (on the group means), and report both. When they disagree, the aggregate number is answering a question about group composition, and it is rarely the question that was asked. See Parkinson’s, section 6 for the between-versus-within table on subjects.
What to do about drift
- Re-anchor the base rate on the most recent period before using probabilities. It is a one-parameter fix and it is usually most of the problem.
- Prefer features that carry the level (lags, recent means, the entity’s own recent history) over features that encode a period.
- Refit on a rolling or expanding window in use, and keep the walk-forward table as the monitoring report.
- Say in the write-up which of the four kinds was found, with the diagnostic that found it. “Distribution shift occurred” is not a finding; “the positive rate rose from 9% to 52% and the model’s mean probability did not” is.