Prediction boundaries: what would actually be known?
The one question that removes most leakage before it happens. At the exact moment the prediction would be made, which columns exist? Sort every column into before, at, after, derived-from-the-future, or unsure, and do it before building a feature.
Every predictive study has a moment at which the prediction is made and used. Most leakage is a column that does not exist yet at that moment. The habit that catches it is not a statistical test; it is a sentence, written down before any feature is built:
At the exact moment this prediction would be made, what information would actually be known?
Then every column in the table gets sorted against that sentence.
The five bins
| Bin | Meaning | Examples | Use it? |
|---|---|---|---|
| Known before | recorded before the moment of prediction, and would stay recorded | the schedule, the customer’s history, yesterday’s value, the booking as made | yes |
| Known at | observable at the moment itself | the current weather, the current load, the airport’s delays so far | yes, if the model runs at that moment |
| Observed after | written after the event the target describes | the call’s duration, the parking space at check-in, the assigned room, the reservation status, a total that includes the outcome | never |
| Derived from the future | computed from rows after the moment, usually by accident | a centred rolling mean, a full-sample z-score, a target-encoded category fitted on all rows, a lag with the wrong sign | never; rebuild it |
| Unsure | the timing is not documented | “booking changes”, “special requests”, “last contact day”, any column that accumulates | investigate; if it stays unsure, leave it out and say so |
The third and fourth bins are the leaks. The fifth is where the honest work is, because the documentation rarely says when a field is written, and the only ways to find out are to ask, or to look for the tell: a column whose relationship with the target is far too clean.
The tell
A post-event column does not look wrong in a summary table. It is numeric or categorical, complete, plausibly scaled. What gives it away is the size of its effect:
- call duration: one column alone scores an AUC of 0.82 on a problem where everything else together scores 0.76;
- parking spaces: zero cancellations out of seven thousand bookings;
- the taxi total: a linear model recovers the tip by subtraction, R² 0.91;
- a centred moving average: a t-statistic of 144 on a daily return signal.
When one column is worth more than a change of model class, it is a suspect. See debugging a model that looks too good.
The boundary moves
The same table can have several boundaries, and the feature set changes with each. The flight-delay study builds three models: the day before (schedule only), two hours before (plus the weather observed then and the delays already happening at the airport), and at departure (not a prediction). The load-forecasting study names two horizons and builds different lags for each. A score without a stated boundary is not a score.
The boundary also decides what the observable version of a feature is. Two hours before departure, a flight scheduled two hours earlier has either left, with a known delay, or is still waiting, with a delay of at least 120 minutes so far. Clipping at 120 turns an after-the-fact column into an honest at-the-moment one. The same move applies to any accumulating quantity: use the value as it stood at the boundary, not the final one.
Things that are usually after the event
- statuses and their dates (
reservation_status,closed_at,outcome) - durations of the thing being predicted
- totals that include the target
- anything recorded at delivery, check-in, settlement, or resolution
- counts of changes, requests or contacts that keep accumulating until the end
- columns whose null pattern matches the target (recorded only when the event happened)
- the universe itself, when it was assembled with hindsight (survivorship)
Things that are usually fine and feel risky
- the customer’s own past outcomes (previous cancellations, previous purchases), as long as they are past
- lagged values of the target, at a lag longer than the horizon
- the current value of an exogenous series (today’s temperature for tomorrow’s load), provided the study says it is an upper bound on what a forecast would give
- calendar columns, unless they encode a campaign rather than a season (bank marketing, section 5)
The habit, in three lines
- Write the moment of prediction into the question.
- Sort every column into the five bins, in a table, before any feature exists. Put the table in the write-up.
- For every derived feature, print three rows with the feature and the target side by side and read the dates.
The pre-analysis protocol has a line for this. The time-boxed workflow puts it in stage 1, because it is cheapest there.