Pre-analysis protocol · print(Purple)

A pre-analysis protocol is the five decisions that determine what a result means, written down before the result exists. It is short on purpose. Five lines, dated, not edited afterwards.

The template

Protocol, fixed on [date] before any feature was built

Target. What exactly am I predicting? (the column, the horizon, the transformation)
Signal. What exactly am I predicting it with? (the column, the window, the normalisation)
Split. Which dates build, which dates check, and the gap between them. (actual dates)
Score. The one number that says whether it worked. (and how its uncertainty is measured)
Baseline. The dumb prediction it has to beat. (often zero; sometimes the boring mechanism)

Rule: anything I try is chosen on the build period and reported on both. I count how many things I try. The check period is reported once.

Why it is frozen

Once a number disappoints, three edits present themselves as reasonable. Move the split a little. Swap the score for one that is kinder to this signal. Redefine the target so that it is closer to what the signal happens to predict. Each is defensible in isolation and together they turn a hold-out into an in-sample fit with extra steps.

The protocol is not there to stop you exploring. It is there so that at the end you can say “I fixed this before I looked”, and mean it. That sentence is worth more than the result it describes. If something in the protocol turns out to be genuinely wrong (the target column was mislabelled, say), change it, date the change, and report both versions.

The code that encodes it

The split and the score are the two lines that most often drift, so write them once and reuse them.

import pandas as pd, numpy as np

df = pd.read_csv("all_stocks_5yr.csv")
df["date"] = pd.to_datetime(df["date"])
df = df.rename(columns={"Name": "ticker"}).sort_values(["ticker", "date"]).reset_index(drop=True)
per = df.groupby("ticker").size()
df = df[df["ticker"].isin(per[per >= 1000].index)].copy()
g = df.groupby("ticker")

# 1. target and 2. signal, built exactly as the protocol says
df["ret"] = g["close"].pct_change()
df["fwd_ret"] = g["ret"].shift(-1)
df["signal"] = -df["ret"]
df = df.dropna(subset=["ret", "fwd_ret"])

# 3. split: chronological, with an embargo, actual dates
cut, embargo = pd.Timestamp("2016-06-30"), pd.Timedelta(days=7)
train = df[df["date"] <= cut]
test = df[df["date"] > cut + embargo]
print("build:", train["date"].min().date(), "->", train["date"].max().date(), "|",
      "check:", test["date"].min().date(), "->", test["date"].max().date())

# 4. score: mean daily cross-sectional rank IC, with t and hit rate
def daily_ic(d, sig, tgt="fwd_ret"):
    return (d.groupby("date")[[sig, tgt]]
             .apply(lambda x: x[sig].corr(x[tgt], method="spearman"))
             .dropna())

def report(ic, label):
    m, s, n = ic.mean(), ic.std(), len(ic)
    print("%-22s mean IC %+.4f   t = %+.2f   hit = %.2f   days = %d" % (label, m, m / (s / np.sqrt(n)), (ic > 0).mean(), n))

# 5. baseline: IC = 0, so the t-statistic is the test
report(daily_ic(train, "signal"), "reversal, build")
report(daily_ic(test, "signal"), "reversal, check")

build: 2013-02-11 -> 2016-06-30 | check: 2016-07-08 -> 2018-02-06
reversal, build        mean IC +0.0114   t = +2.18   hit = 0.53   days = 854
reversal, check        mean IC +0.0156   t = +1.98   hit = 0.55   days = 399

On intraday data use < cut and >= cut + embargo so each side gets whole days. kit.split does this.

The embargo of a week means that, for this one-day target, no build-period row’s target falls inside the check period. See chronological splits with an embargo for why the gap matters, how wide it should be, and why it is better counted in trading days than calendar days.

Two filled-in examples

Short-term reversal

Target. Tomorrow’s close-to-close return, per ticker.

Signal. Minus today’s close-to-close return.

Split. Build to 2016-06-30. Check from 2016-07-08. One-week embargo. Check reported once.

Score. Mean daily cross-sectional Spearman IC, with t-statistic across days and hit rate.

Baseline. IC of zero.

Case study →

Relative volume and move size

Target. |tomorrow’s return| ÷ trailing 20-day standard deviation of returns, per ticker. “How big was tomorrow, relative to normal for this stock.”

Signal. Today’s volume ÷ trailing 20-day mean volume, per ticker.

Split. Same dates and embargo as the reversal study.

Score. Mean daily cross-sectional Spearman IC, with t.

Baseline. Today’s normalised move size, |today’s return| ÷ the same 20-day vol. Volatility clusters, so busy days are volatile days and volume can look predictive while adding nothing. The IC is computed for the signal, for the baseline, and for both together.

Case study →

The second example shows the protocol doing its most useful job: the baseline is not zero but a specific boring mechanism, and the protocol commits to scoring it before there is any temptation not to.

What counts as a change

Moving the split date, widening or narrowing the embargo.
Changing the score (Pearson to Spearman, IC to a decile spread) after seeing a number.
Redefining the target’s horizon or normalisation.
Dropping a subset of rows that happens to help.

What does not count: building a new candidate signal and testing it under the same protocol. That is the hypothesis round, and the only requirement is that the candidate is chosen on the build period and that you count it. See selection bias from the inside.