Questions to resolve before modelling

A vague brief has several readings, and the cheapest moment to find out which one was meant is before any modelling. Whoever set the question also knows things the table does not: what a column really means, whether the extreme moves are real, whether a small honest number would be useful or a disappointment.

So the first pass over the data ends with a short, structured conversation. It should take about ninety seconds to say.

The summary, in order

1. How I have read the question. Cross-sectional (“which groups do better than others?”) or single-series (“does the aggregate go up?”). At what horizon. Five hundred stocks a day is a lot of evidence; one market series is not, and the two need different repeats.
2. The measure and its baseline. “Rank correlation between today’s signal and tomorrow’s outcome, one per day, against a baseline of zero.” Or whatever the protocol says.
3. Three things wrong with the data and what I am doing about each. The output of the interrogation, as decisions.
4. The hold-out I am about to fix. Which dates, with what gap, and that it will be reported once.
5. Two questions back. Is that the question you meant? and Is a small honest number an acceptable answer?

The second question matters. If the honest answer is going to be “there is a small effect that would not survive costs”, it is better to know now that this is a useful finding rather than a failure.

The pushback to expect

Whoever set the question will not usually correct the reading. They will push for decisions where the summary offered descriptions.

What you said	What you will be asked
“Twenty-two tickers have short histories.”	Drop them or keep them? Either is fine. A decision, not a shrug.
“There are forty-six moves over 25%.”	Are they real or unadjusted splits? Check two by eye before deciding.
“Volume relative to the stock’s own history.”	Exactly how? Over its trailing average, a z-score, or a rank within the ticker? Pick one.
“The size of tomorrow’s move.”	Absolute return? High minus low over close? Raw, or relative to how much that stock normally moves? A stock that moves 3% a day will always look big next to one that moves 0.5%.
“High volume today predicts a big move tomorrow.”	What is the boring explanation you have to rule out first? What would you compare against to show volume adds anything?

The last row is the important one. For almost every interesting signal there is a dull mechanism that produces the same pattern. Busy days are volatile days, so today’s volume “predicting” tomorrow’s volatility may be nothing more than volatility clustering with extra steps. The protocol has to name the boring explanation and score it alongside the interesting one. See does a signal add anything beyond a baseline? and the relative volume case study, where the pushback above was answered.

A filled-in example

For the short-term reversal study, the summary was:

I have read the question as cross-sectional, which stocks beat which tomorrow, rather than predicting the market. I am using rank correlation between today’s signal and tomorrow’s return as the measure, one per day, with zero as the baseline. Three things about the data: it is the end-of-period index constituents, so it is survivorship-biased and I will flag that on every result; twenty-two names have short histories and I have dropped them; and there are about forty extreme single-day moves that I am keeping but neutralising with ranks. I am going to fix a hold-out now, the last nineteen months with a one-week gap, and not touch it until the end. Is that the question you meant, and is a small honest number an acceptable answer?

Two things happen. If the question was misread, it gets corrected while the correction is free. And the person who set it now knows the analysis was designed before any result was seen.

Checklist