Analysing a new dataset in a fixed block of time

The failure mode of a time-limited analysis is not running out of ideas. It is spending the first two-thirds of the time on a model that turns out to be leaking, and the last third discovering it. The defence is to have a presentable result early and to make every later step an improvement you can measure against it. The stages below are in the order that achieves that. The percentages are a default; the shape matters more than the numbers.

For open-ended research questions on a panel the research workflow is the more specific version of this page. For finding out which kind of problem you have in the first place, start at I opened a dataset. What now?

Stage 1: orient (15%)

The output of this stage is a short list of facts and decisions, not a clean table.

• Load it. Shape, dtypes, a few rows read as sentences.
• What is one row? Say it out loud.
• What is the target? Its distribution, its base rate if binary, its tail if continuous.
• Is there a clock? Range, gaps, duplicates on the time key. Plot the monthly mean of the target.
• Do entities repeat? Rows per entity; how much of the target’s variance is between entities.
• Nulls per column, exact duplicates, cardinality of every categorical.
• The prediction boundary. For each column: known before the prediction, known at it, or observed after. Post-event columns are excluded now, not discovered later. → prediction boundaries
• The validation design, chosen from the structure, written down. → choosing a split

Stop when you can fill in the five-line protocol: target, features, split, score, baseline. Do not model before this exists.

Stage 2: baselines (10%)

Deliberately simple predictions, scored under the protocol. At least two:

Problem	Baselines
regression	train mean; train median; a one-feature linear fit
classification	majority class; the constant base-rate probability; a one-feature logistic fit
forecasting	last value; same time last cycle; train mean by calendar cell
repeated entities	the entity’s own train mean or last value

The baseline is the number every later number is compared against, and it is the number in the summary if nothing else works. Getting it also proves the pipeline runs end to end: file to scored hold-out. → building baselines

Stage 3: look (10%)

Exploration with a purpose: each plot is chosen because its answer changes a modelling decision.

• Target against the two or three features most likely to matter, as binned means rather than raw scatter.
• Target over time, if there is a clock.
• Target by the categories with the fewest levels.
• The residuals of the baseline, if it has any structure worth naming.

Skip anything that is “interesting” but would not change what you build. → evaluating signals for the binned-scatter habit.

Stage 4: the first real model (15%)

The simplest model that can test the hypothesis from stage 3. Linear or logistic regression on a handful of features, or a shallow tree ensemble if the interactions in stage 3 were obvious. Fitted on the training rows only, with every transformation (scaling, encoding, imputation) learned on the training rows only, scored once on the hold-out under the protocol’s metric, and compared with the baseline.

At the end of this stage there is a result that can be written up. Everything after it is optional.

Stage 5: diagnose (15%)

This stage decides what stage 6 is allowed to do.

• Residuals: against the fitted value, against time, against the largest categories. A residual with a sign for months on end is a missing level, not noise.
• Subgroups: the metric by category, by entity size, by period. Where is it worst?
• Calibration, for probabilities: predicted versus observed rate by decile.
• The largest errors, individually. Read ten of them.
• Feature behaviour: the one or two features doing most of the work. If one dominates, ask when it is recorded.
• The random-split check: if you used a chronological or grouped split, run the shuffled version once. If it is far better, you have found how much leakage was avoided, and it belongs in the write-up.

→ debugging a model that looks too good, classification metrics

Stage 6: improve selectively (15%)

Only what the diagnostics motivated. One change at a time, each scored under the same protocol:

• a transformation of the target if the residual variance grew with the fitted value;
• an interaction or a nonlinear model if a binned plot bent;
• lag or rolling features if the residuals had a level;
• a different encoding if a categorical was the problem;
• regularisation if the training score was far above the hold-out score.

Count what you tried. Two changes that each helped are a result; ten changes of which one helped is a selection problem.

Stage 7: try to break it (10%)

The result is not a result until it has survived an honest attempt to kill it.

• Does it hold on the second half of the hold-out period, or on the entities you have not looked at?
• Does it survive removing the rows you were worried about in stage 1?
• Does it survive removing the one feature that does most of the work?
• Is it materially better than the baseline, or merely numerically better?
• Would every feature really exist at prediction time?

→ attacking your own result

Stage 8: communicate (10%)

A hard stop, after which only the write-up is touched. Seven things, one sentence and one number or chart each:

1. the question as you read it;
2. the data, and the three things wrong with it;
3. what would be known at prediction time, and what you excluded because of it;
4. the validation design and why;
5. the baseline, the model, and the gap between them;
6. the strongest finding and the biggest limitation;
7. what you would do with more time.

→ presenting a quantitative analysis

Stopping rules

• No model before the protocol exists. Stage 1 is where the leaks are found cheaply.
• No stage 6 without stage 5. Complexity added without a diagnosis to justify it is guesswork with a longer runtime.
• The write-up starts at stage 4. It is edited from then on, never started at the end.
• A defensible simple result beats an elaborate model whose behaviour you cannot explain. If the last change made the number better and you do not know why, revert it and say so.
• When the clock says stop, stop. The last hour spent chasing a better number is the hour that was needed to explain the one you have.