Choosing a train/test split
The split has to follow the structure of the data, not the default of the library. A decision table by structure, the one diagnostic that says which axis to hold out, and the habit of running the shuffled split once as a control.
A hold-out estimates how the model would do on rows it has not seen. What “not seen” means depends on what the rows share. Rows that share a date share that day’s weather, market move or pollution episode; rows that share an entity share that entity’s level; rows that are exact copies share everything. A shuffled split puts one of each pair on either side of the line and then rewards the model for recognising the other. The split has to be chosen so that the test rows share nothing with the training rows that they would not share in use.
By structure
| The data | The honest split | Because | Example |
|---|---|---|---|
| static rows, one per entity, no clock | shuffled, stratified on a rare target | nothing to leak, as long as there are no near-duplicates | house prices (split by sale year anyway, since one existed) |
| a single time series | chronological, with an embargo of at least the target horizon | the level and the neighbours leak | load forecasting |
| a panel: entities × time | chronological; hold out entities as well if the question is about new ones | dates leak across entities; entities leak across dates | twelve sites, one sky, chronological splits with an embargo |
| repeated measurements of entities, question about new entities | group hold-out on the entity (GroupKFold) | the model learns identities | Parkinson’s subjects |
| repeated measurements, question about known entities | within-entity chronological: early rows train, late rows test | the entity’s history is a legitimate feature, its future is not | same page, design 2 |
| events with a clock (bookings, calls, trips) | chronological on the event time | base rates and populations drift | bank marketing, hotel cancellations |
| anything with exact or near duplicates | deduplicate first, then the rule above | twins across the line are memorised | hotel cancellations, section 3 |
“Chronological” always means: choose the cut on the calendar before looking at results, embargo the boundary by the target horizon, and look at the test period once. See chronological splits with an embargo and selection bias from the inside.
The diagnostic
When it is not obvious which axis leaks, decompose the target’s variance:
share explained by entity = var( target's mean per entity ) / var( target )
share explained by date = var( target's mean per date ) / var( target )Two lines of pandas (Parkinson’s, section 1; Beijing, section 2). The axis with the variance is the axis a shuffled split leaks along, and the axis to hold out. In the patient data it was 93% entity and the honest split was by subject; in the air-quality data it was 94% date and holding out subjects changed nothing.
For a plain time series, the equivalent is the mean of the target by year. If it moves, a shuffled split will report a number that is not a forecast (bike rentals). If it does not, the shuffled number may be honest by accident (traffic volume), and the chronological one is still the one to report.
Run the shuffled split once, as a control
After the honest split has produced its number, fit the same model under a shuffled split and report both. The gap is the amount of leakage the honest design avoided, and it is a useful number: a gap of 0.86 against 0.63 tells the reader how much a naive analysis would have overstated the result, and a gap of zero tells them the series is stationary. Either way the shuffled number is the control, not the result.
What the split does not fix
- Preprocessing fitted on all rows. Scalers, encoders, imputers and target encodings learned on train and test together leak the test set’s statistics into the features. Fit them on the training rows only; see portable pandas idioms for the one-hot form.
- Feature selection on the test set. Choosing the best of twenty candidates by their test score is a selection on the test set, whatever the split; see selection bias from the inside.
- Post-event columns. No split makes a column that is written after the outcome honest; see prediction boundaries.
- Too few groups. Holding out three of twelve stations or eight of forty-two subjects gives a noisy estimate; report the per-fold spread, not just the pooled number.