Missing values: classify before you treat
A blank means one of three things, and each needs a different treatment. The checks that tell them apart, whether absence itself predicts the target and how to test that claim honestly, what each treatment costs, and where imputation leaks.
df.isna().mean() says how much is missing. It does not say why, and the why decides everything that follows. This page is the reference version of the house-price study, where every step below is run on a real table.
Three kinds of blank
| Kind | Meaning | How to recognise it | Treatment |
|---|---|---|---|
| structural | the thing does not exist (no garage, no previous campaign, no agent) | the blank lines up exactly with a zero in a related size or count column | its own category (“None”) and zero for the sizes; never drop the rows |
| not applicable | the question did not arise (pool quality for houses without a pool) | mostly missing, and missing for a definable subset | same as structural |
| not recorded | the value exists and was not captured | missingness varies with other columns, or with the period, or with who entered the data | impute, with an indicator, and check what the missingness tracks |
Sentinels are a fourth kind in disguise: a −1 in a days-since column, a 0 in a field that cannot be zero, “unknown” as a category level, 9999 anywhere. A linear model reads −1 as a small number of days. Find them with the physical-range check and recode them as blanks before deciding which kind they are. See bank marketing (pdays = -1, four “unknown” columns) and leaked columns and labels that aren’t labels (a zero tip that meant “not recorded”).
The checks
- Share missing per column, sorted. Anything above 90% is probably structural or not-applicable.
- The pairing check. For each mostly-missing quality column, find the size or count column that goes with it and confirm the blanks line up with zeros. If they line up exactly, the blank is structural.
- Missingness by group. The share missing by category, by period, by source. A column missing at 49% on cul-de-sac lots and 13% elsewhere is not missing at random, and imputing it with a global median will be wrong in a patterned way.
- Missingness against the target. The target’s mean when the column is blank against when it is present. If they differ, absence carries information; whether that information is about the target or about something the table already has is the next check.
- Conditioned on the obvious confounder. Repeat check 4 within the grouping column that check 3 found. In the house data, the 7% price premium of missing frontage was zero within neighbourhoods. See grouped confounding.
- Rows with any gap among the columns you will use. This is the share of the data a “drop rows” policy throws away, and it is often most of it.
What each treatment costs
- Dropping rows is the expensive option: it discards every row with a structural blank (every house without a fireplace), shrinks the training set, and changes the population the model describes. In the house data it lost 60% of the rows and raised the error by 14%.
- Dropping columns because they are “mostly missing” discards structural information that may be the only record of a fact.
- Median or mode imputation with a “Missing” category is the safe default and, on a well-designed table, as good as anything more elaborate.
- Missing indicators are cheap insurance: harmless when redundant, and the only thing that helps when absence is the only record of the fact. Test which case you are in by removing the correlated columns and seeing how much the indicator recovers.
- Group-wise imputation (the neighbourhood median rather than the global one) is the right idea when check 3 found a pattern, and worth the four extra characters even when, as in the house data, it does not change the score.
- Model-based imputation is rarely worth its complexity in a short study, and every fitted imputer is one more thing to fit on training rows only.
Where imputation leaks
Every imputation value is a statistic learned from data. The median, the group medians, the mode, the fitted imputer: all of them are learned on the training rows and applied to the test rows, exactly like a scaler. A median computed on the whole table is a small leak; a group median computed on the whole table is a larger one; an imputer fitted on the test rows is the test set in the features. See choosing a split.
A subtler leak: missingness that is caused by the outcome. A column recorded only when the event happened (a delivery time that is blank for undelivered orders, a resolution date blank for open cases) is a post-event column wearing a missing-indicator costume. Its blank pattern is the target. See prediction boundaries.
What to write down
For each column with more than a few percent missing: the kind, the check that established it, the treatment, and whether absence tracked the target after conditioning. Three columns of a table, and the sentence “missing values were handled” becomes something a reader can verify.