print(Purple)
Home/Research Workflows/A research workflow for open-ended questions

A research workflow for open-ended questions

The stages between receiving an unfamiliar table with a vague question and presenting an honest number. Interrogate, agree the question, fix the protocol, get a baseline, try one idea, attack it, stop.

Updated 23 Sept 20261 code block verifiedworkflowprotocolbaselinehold-outpresentation

Most research questions arrive vague. “Here is a table. Have a look and tell me whether there is anything predictable about X.” The first job is not to model anything. It is to turn that sentence into a question that a number can answer, and to write down every choice made on the way so that the choices can be defended later.

This page is the spine. Each stage links to a page with the detail, and the case studies show the whole shape end to end on public data.

The shape

  1. Interrogate the table. Not clean it. Find the three things that look wrong.
  2. Agree the question. Say how you have read it, what you will measure, and what you are doing about the three problems. Ask whether that is the question meant.
  3. Freeze the protocol. Target, signal, split, score, baseline. Five lines, on paper, before any feature.
  4. Get a baseline number. The simplest thing that can score. If it cannot, the pipeline is broken and you have found out cheaply.
  5. One hypothesis round. A mechanism, chosen on the training period, reported on both.
  6. Attack your own result. Is it stable? Could anyone use it?
  7. Stop and present. Title, one sentence, one number or chart per slide.

Roughly a quarter of the available time goes on stages 1 to 3, a quarter on stage 4, a quarter on 5 and 6, and the rest on 7. The presentation exists from the moment stage 4 produces a number. Everything after that is an improvement you can measure, not a rescue.

1. Interrogate, don’t clean

A table can be wrong in only four places: the rows, the axis, the values, and the derived quantities the question needs. Plus one domain question, which is what this kind of data habitually lies about. Each check is one pandas line. Pick the three findings that came back wrong and write them down.

Those three findings are not a side-note. They are the agenda for the conversation with whoever set the question. Do not treat “I have checked the data” as a stage that ends when the checks are done; it ends when the findings have been turned into decisions.

→ First-pass dataset interrogation · Common data-quality traps by dataset type

2. Agree the question

Ninety seconds, out loud or in writing: how you have read the question (cross-sectional or single-series, at what horizon), the measure and its baseline, the three problems and your decision on each, the hold-out you are about to fix. Then two questions back: is that the question you meant? and is a small honest number an acceptable answer?

Expect to be pushed for decisions rather than descriptions, for exact definitions of both the thing predicted and the thing predicting it, and for the boring explanation you have to rule out first.

→ Questions to resolve before modelling

3. Freeze the protocol

Five lines, written before a single feature is built and not changed afterwards:

Targetwhat exactly am I predicting?
Signalwhat exactly am I predicting it with?
Splitwhich dates build, which dates check, and the gap between them
Scorethe one number that says whether it worked
Baselinethe dumb prediction it has to beat

It is frozen because the temptation later is to move the split, swap the score or redefine the target once the number disappoints. Being able to say “I fixed this before I looked” is worth more than the result.

→ Pre-analysis protocol · Chronological splits with an embargo · Evaluating predictive signals

4. Get a baseline number

The simplest signal that can score under the protocol. For a cross-sectional return question that is usually a one-line signal and a daily rank correlation. The whole pipeline, from file to scored hold-out, fits in thirty lines:

import pandas as pd, numpy as np

df = pd.read_csv("all_stocks_5yr.csv")
df["date"] = pd.to_datetime(df["date"])
df = df.rename(columns={"Name": "ticker"}).sort_values(["ticker", "date"]).reset_index(drop=True)

# decision from interrogation: drop the 22 tickers with short histories
per = df.groupby("ticker").size()
df = df[df["ticker"].isin(per[per >= 1000].index)].copy()

g = df.groupby("ticker")
df["ret"] = g["close"].pct_change()          # today's close-to-close return
df["fwd_ret"] = g["ret"].shift(-1)           # TARGET: tomorrow's return. shift(-1) looks forward
df["signal"] = -df["ret"]                    # SIGNAL: minus today's return (short-term reversal)
df = df.dropna(subset=["ret", "fwd_ret"])

cut, embargo = pd.Timestamp("2016-06-30"), pd.Timedelta(days=7)
train = df[df["date"] <= cut]
test = df[df["date"] > cut + embargo]

def daily_ic(d, sig, tgt="fwd_ret"):
    """One Spearman correlation per date, across every ticker that date."""
    return (d.groupby("date")[[sig, tgt]]
             .apply(lambda x: x[sig].corr(x[tgt], method="spearman"))
             .dropna())

def summary(ic):
    m, se = ic.mean(), ic.std() / np.sqrt(len(ic))
    return {"mean_ic": round(m, 4), "t": round(m / se, 2), "hit": round((ic > 0).mean(), 2), "days": len(ic)}

print("train", summary(daily_ic(train, "signal")))
print("test ", summary(daily_ic(test, "signal")))
train {'mean_ic': 0.0114, 't': 2.18, 'hit': 0.53, 'days': 854}
test  {'mean_ic': 0.0156, 't': 1.98, 'hit': 0.55, 'days': 399}

That is a result. Read it the way you would say it: “the daily rank IC of one-day reversal is about 0.015 out of sample, t just under 2, positive on 55% of days. Small, and real-looking in the sense that it is consistent across two independent periods.” From here on there is something to present.

5. One hypothesis round

Mechanism first, then test. “A week of losses bounces harder than a day” is a mechanism; “let me try five-day returns” is not. The candidate is chosen on the training period and reported on both periods. Count how many things you tried, because the count is part of the result.

The honest sentence at the end of this stage is often: I tried two things. The simpler one held; the one that looked better in-sample didn’t. Say exactly that. It is the sentence that tells a reader you can be trusted with a hold-out.

→ Selection bias from the inside

6. Attack your own result

Two attacks, always the same two.

  • Is it stable? Split the hold-out into halves. Plot the cumulative IC. Compute the IC by year.
  • Could anyone use it? Long the top decile, short the bottom. Turnover. Break-even transaction cost.

Then the sensitivity checks that the interrogation suggested: does the result survive removing the extreme rows, and does it look the same in the large names and the small ones?

→ Attacking your own result

7. Stop and present

A hard stop, well before the deadline, after which only the presentation is touched. Seven slides: the question as you read it; the data and three things wrong with it; how you decided what counts; the baseline result; what else you tried; does it hold and could you use it; what you would do with another day. Then a prepared list of the questions you will be asked.

→ Presenting a quantitative analysis · The last three slides

Two complete passes