Debugging a model that looks too good
A checklist for the moment a number comes out better than the problem deserves. The size of the jump is the first clue; then the seven ways a score gets inflated, in the order they are worth checking, each with the worked example where it happened.
A result that is too good is more common than a result that is good, and it is found by the same person who produced it far less often than by the next person to look. The checks below are cheap, and each one has a worked example on this site where it would have caught a real inflated number. Run them before presenting anything, and run them again on any number someone else presents to you.
First: how big is the jump?
Every problem has a rough ceiling. A day-ahead sign of a stock return does not score an AUC of 0.7; an hourly count with a strong daily cycle does not score an R² of 0.99 a day ahead without lags; a marketing conversion with demographic features does not score 0.9. When the number clears the ceiling, the question is not “what did we do right”.
The most useful calibration is the gap between model classes. On an honest feature set, a forest or boosted ensemble beats a logistic or ridge regression by a few points. When one column, one split or one preprocessing step is worth fifteen, it is the column, the split or the step, not the model. See bank marketing, section 3: boosting lifted an honest 0.76 to 0.79; a leaked column lifted it to 0.91.
The seven checks, in order
1. Is a column written after the outcome?
For each feature, ask when it is recorded relative to the moment of prediction. Durations, statuses, totals that include the target, anything recorded at check-in, settlement or resolution. The tell is a single column whose effect dwarfs everything else. → prediction boundaries, call duration, parking spaces and assigned rooms, the taxi total
2. Is a column an arithmetic function of the target?
Regress the target on each column alone, or correlate; anything near ±1 is the target in disguise. Components that sum to the target (casual + registered = cnt) count. → leaked columns, bike rentals, section 1
3. Does the split let the model see the test period?
On anything with a clock, print the mean of the target by year. If it moves, a shuffled split leaks the level, and the honest number is the chronological one. Then check the boundary: do the last training rows’ targets reach into the test period? → the random split that looked great, chronological splits with an embargo
4. Does the split let the model see the test entities, or the test dates through other entities?
Count rows per identifier. If entities repeat, decompose the target’s variance by entity and by date; the axis with the variance is the axis the shuffled split leaks along. Hold it out and re-score. → a model that learns who, not how ill, twelve sites, one sky, choosing a split
5. Are there duplicates across the line?
df.duplicated().mean(), then the same on the natural key. Twins in train and test turn any model into a lookup table. A striking single-column pattern that lives mostly in duplicated rows is a pattern about data entry. → hotel cancellations, section 3
6. Was anything fitted on all the rows?
Scalers, imputers, one-hot column lists, target encodings, feature selection, rolling normalisation over the full sample, a “smoothed” series. Each one carries test statistics into training features. Refit every transformation on training rows only and re-score; then check that no rolling or centred window reaches forward. → forward targets and look-ahead leakage, target-encoded aircraft, portable idioms
7. Was the winner chosen on the test set?
Count the candidates tried. The best of twenty on the same sample has a t-statistic near 2 by construction. If the test period was looked at more than once, say how many times. → selection bias from the inside, pre-analysis protocol
The three-line confirmation
When a check fires, the fix has to be measured, not asserted:
- Remove the column, or change the split, or refit the preprocessing.
- Re-score under the same protocol.
- Report both numbers, and the gap. The gap is the finding.
When nothing fires
Sometimes the number is right. The traffic-volume forest scored 0.95 under both a shuffled and a chronological split, and the reason was visible in the yearly means: the series did not drift, so there was no level to leak, and the calendar really does explain that much of a road’s traffic. The checklist does not exist to make good results go away; it exists so that when one survives it, the sentence “I checked” means something specific. → traffic volume against a seasonal baseline, attacking your own result