Diagnostics
Debugging a model that looks too good, what each kind of dataset lies about, leaked columns, missing values, drift, and whether an extreme value is real.
Debugging a model that looks too good
A checklist for the moment a number comes out better than the problem deserves. The size of the jump is the first clue; then the seven ways a score gets inflated, in the order they are worth checking, each with the worked example where it happened.
Common data-quality traps by dataset type
The four generic checks are the same for every table. What changes is what each kind of data habitually lies about. One row per dataset type, with the lie, the tell, and the quantities you would build.
Extreme values: real events or artefacts?
Forty-six daily moves over 25% in a file of 619,040 rows. How to tell a bad print from a split from a real event by looking at two of them, and the three policies available once you have decided.
Leaked columns and labels that aren't labels
A column that is an arithmetic function of the target, and a zero that means "not recorded" rather than "zero". Two checks that take a minute and save a whole study, shown on a public taxi-trip table.
Distribution shift: performance gets worse over time
A model that worked and then stopped is usually the world moving, not the code breaking. The four things that drift, the diagnostics that tell them apart, what each does to the score and to the probabilities, and the related trap where the aggregate relationship differs from the one inside every group.
Missing values: classify before you treat
A blank means one of three things, and each needs a different treatment. The checks that tell them apart, whether absence itself predicts the target and how to test that claim honestly, what each treatment costs, and where imputation leaks.