Chronological splits with an embargo

The purpose of a hold-out is to estimate how a result would have looked to someone who did not know the future. On time-ordered data that means the check period has to come after the build period, with nothing crossing the boundary in either direction.

Why not a random split

Rows on a panel are not exchangeable. Every stock on a given date shares that date’s market move, news and volatility. A random split puts three-quarters of a date’s rows in the build set and a quarter in the check set, so a model can learn what happened on that date from the build rows and be rewarded for it on the check rows. Nothing about that is available in real time.

Even a single-signal study with no fitted model is better off with a time split, because the point of the check period is to see whether the effect persisted, and a random split cannot answer that.

The split

Two decisions: the cut date and the embargo. Write both down in the protocol before any feature exists.

import pandas as pd, numpy as np

df = pd.read_csv("all_stocks_5yr.csv")
df["date"] = pd.to_datetime(df["date"])
df = df.rename(columns={"Name": "ticker"}).sort_values(["ticker", "date"]).reset_index(drop=True)
g = df.groupby("ticker")
df["ret"] = g["close"].pct_change()
df["fwd_ret"] = g["ret"].shift(-1)
df["fwd_ret5"] = g["ret"].transform(lambda x: x.shift(-5).rolling(5).sum())

cut = pd.Timestamp("2016-06-30")             # roughly two-thirds build, one-third check
embargo = pd.Timedelta(days=7)               # calendar days: simple, and checked below
train = df[df["date"] <= cut]
test = df[df["date"] > cut + embargo]
print("build: %s -> %s  (%d rows, %d days)" % (train["date"].min().date(), train["date"].max().date(), len(train), train["date"].nunique()))
print("check: %s -> %s  (%d rows, %d days)" % (test["date"].min().date(), test["date"].max().date(), len(test), test["date"].nunique()))

build: 2013-02-08 -> 2016-06-30  (416612 rows, 855 days)
check: 2016-07-08 -> 2018-02-07  (200436 rows, 400 days)

How wide the embargo has to be

The leak the embargo prevents is specific: the targets of the last build rows are computed from returns that fall inside the check period. A one-day target needs a one-trading-day gap; a five-day target needs five. Features that look back across the boundary are fine, because they use past data, which is what they would have used in real time.

Check it rather than assume it. The last date any build row’s target reaches into must be strictly before the first check date:

last_build_day = train["date"].max()
trading_days = pd.DatetimeIndex(np.sort(df["date"].unique()))
pos = trading_days.get_loc(last_build_day)
for h in (1, 5):
    reach = trading_days[pos + h]                    # an h-day target on the last build day uses the h trading days after it
    print("%d-day target on the last build day reaches %s | first check day %s | %s"
          % (h, reach.date(), test["date"].min().date(), "ok" if reach < test["date"].min() else "OVERLAP"))

1-day target on the last build day reaches 2016-07-01 | first check day 2016-07-08 | ok
5-day target on the last build day reaches 2016-07-08 | first check day 2016-07-08 | OVERLAP

Seven calendar days after Thursday 30 June 2016 is Friday 8 July, and the five trading days after 30 June are 1, 5, 6, 7 and 8 July, because the market was closed on the 4th. A “week” of calendar days was four trading days, and a five-day target on the last build row shares a day with the check period. For the one-day target used in the reversal study the week is ample; for the five-day candidate in the same study it was one day short, which is worth knowing even though a one-day overlap on one row out of 400 days changes nothing measurable.

The fix is to count the embargo in trading days, from the file’s own calendar:

h = 5
first_check = trading_days[pos + h + 1]              # the first day no build-period target can touch
test_td = df[df["date"] >= first_check]
print("embargo of %d trading days: check starts %s (the 7-calendar-day version started %s)" % (h, first_check.date(), test["date"].min().date()))

embargo of 5 trading days: check starts 2016-07-11 (the 7-calendar-day version started 2016-07-08)

For a 20-day target, twenty trading days. For a target defined by publication dates, the longest publication lag. Always in the units the target is measured in.

Where to put the cut

Two-thirds build and one-third check is a reasonable default with five years of daily data: enough check period to split into halves for a stability attack, enough build period to choose between candidates. Whatever the choice, it is made once, on the calendar, not on the results.

Reported once

The check period is looked at once, at the end, for whatever the protocol said would be reported. Everything else, every candidate signal, every parameter, every “let me just see”, happens on the build period. Then count how many things were tried, because the count is part of the result. See selection bias from the inside.

If the check period is looked at twice, the second look is in-sample. It is not ruined, but the report has to say “I looked at the hold-out twice”, and the second result carries less weight than the first.

Expanding and rolling re-estimation

For a study with a fitted model and time to spare, the natural extension is walk-forward: fit on years one to three, score year four; fit on one to four, score year five. It gives several out-of-sample looks instead of one and mimics how the model would be used. It costs a few times the compute and is worth it only once the single-split version has produced a number worth refining.

# sketch: yearly walk-forward with an expanding build window
for year in (2016, 2017):
    build = df[df["date"].dt.year < year]
    check = df[df["date"].dt.year == year]
    # fit on build, score on check, collect the per-year score

Summary

Decision	Rule
Split type	by date, never random, on any time-ordered panel
Cut	on the calendar, before any result; about two-thirds build
Embargo	at least the target horizon, counted in trading days from the file’s own calendar, and checked
Check period	reported once; count what was tried on the build period