print(Purple)
Home/Validation/Selection bias from the inside

Selection bias from the inside

Picking the candidate that looked better on the build period never feels like cheating. It feels like picking the better one. A real example where it would have been wrong, and a simulation of how often it is.

Updated 21 Sept 20262 code blocks verifiedselection-biasmultiple-comparisonshold-outwinner's-curse

Two candidate signals, scored on the build period. One looks better. You present it. That is selection bias, and from the inside it does not feel like a mistake; it feels like judgement.

The real example

In the reversal case study, the first signal was minus today’s return. The hypothesis round tried minus the last five days’ return, on the reasoning that a week of losses might bounce harder than a day.

import pandas as pd, numpy as np

df = pd.read_csv("all_stocks_5yr.csv")
df["date"] = pd.to_datetime(df["date"])
df = df.rename(columns={"Name": "ticker"}).sort_values(["ticker", "date"]).reset_index(drop=True)
per = df.groupby("ticker").size()
df = df[df["ticker"].isin(per[per >= 1000].index)].copy()
g = df.groupby("ticker")
df["ret"] = g["close"].pct_change()
df["fwd_ret"] = g["ret"].shift(-1)
df["sig1"] = -df["ret"]
df = df.dropna(subset=["ret", "fwd_ret"])
cut, embargo = pd.Timestamp("2016-06-30"), pd.Timedelta(days=7)
train = df[df["date"] <= cut].copy()
test = df[df["date"] > cut + embargo].copy()
for d in (train, test):                       # the candidate, built inside each period as in the case study
    d["sig5"] = -d.groupby("ticker")["ret"].transform(lambda x: x.rolling(5).sum())

def daily_ic(d, sig, tgt="fwd_ret"):
    return d.groupby("date")[[sig, tgt]].apply(lambda x: x[sig].corr(x[tgt], method="spearman")).dropna()

def report(ic, label):
    m, s, n = ic.mean(), ic.std(), len(ic)
    print("%-24s mean IC %+.4f   t = %+.2f   hit = %.2f   days = %d" % (label, m, m / (s / np.sqrt(n)), (ic > 0).mean(), n))

for sig in ("sig1", "sig5"):
    report(daily_ic(train.dropna(subset=[sig]), sig), sig + ", build")
    report(daily_ic(test.dropna(subset=[sig]), sig), sig + ", check")
sig1, build              mean IC +0.0114   t = +2.18   hit = 0.53   days = 854
sig1, check              mean IC +0.0156   t = +1.98   hit = 0.55   days = 399
sig5, build              mean IC +0.0152   t = +2.88   hit = 0.53   days = 850
sig5, check              mean IC +0.0081   t = +1.04   hit = 0.51   days = 395

On the build period the five-day version was better on every column: higher IC, t of 2.9 against 2.2. With one period, it is the one anybody would have chosen. On the check period it is worse than the one-day version and not distinguishable from zero.

Neither signal “failed”. Two noisy estimates were compared, and the comparison went one way in one period and the other way in the next, which is what noise does. The only defence was that the check period had been fixed before either number was seen.

How often this happens: a simulation

Suppose NN candidate signals all have the same true daily IC of zero, and each day’s measured IC is that plus noise with the standard deviation observed in the real data (about 0.15). Choose the best on 850 build days; score it on 400 check days.

rng = np.random.RandomState(0)
sd, n_build, n_check, trials = 0.15, 850, 400, 2000

for n_candidates in (1, 2, 5, 20):
    build_t, check_t = [], []
    for _ in range(trials):
        build = rng.normal(0, sd, size=(n_candidates, n_build))
        check = rng.normal(0, sd, size=(n_candidates, n_check))
        best = build.mean(axis=1).argmax()                         # chosen on the build period
        build_t.append(build[best].mean() / (sd / np.sqrt(n_build)))
        check_t.append(check[best].mean() / (sd / np.sqrt(n_check)))
    build_t, check_t = np.array(build_t), np.array(check_t)
    print("%2d candidates: build t of the winner, median %+.2f, share > 2: %.2f | check t of the winner, median %+.2f, share > 2: %.2f"
          % (n_candidates, np.median(build_t), (build_t > 2).mean(), np.median(check_t), (check_t > 2).mean()))
 1 candidates: build t of the winner, median +0.01, share > 2: 0.03 | check t of the winner, median -0.04, share > 2: 0.02
 2 candidates: build t of the winner, median +0.57, share > 2: 0.04 | check t of the winner, median +0.02, share > 2: 0.02
 5 candidates: build t of the winner, median +1.12, share > 2: 0.10 | check t of the winner, median +0.02, share > 2: 0.03
20 candidates: build t of the winner, median +1.82, share > 2: 0.38 | check t of the winner, median +0.00, share > 2: 0.02

With twenty candidates that all do nothing, the winner’s build-period t-statistic has a median of 1.8 and clears 2 in nearly four runs out of ten. Its check-period t is centred on zero, exactly as it should be, because the check period was not used to choose it. That is the whole argument for a hold-out in one table.

The rules that follow

  1. Choose on the build period, report on both. Never choose on the check period.
  2. Count what you tried. A t of 2 after two candidates and a t of 2 after twenty are different results. Say the number.
  3. Prefer the simpler candidate on a tie. Or on a small win. The simpler one has fewer ways to have fitted noise.
  4. Say the honest sentence. I tried two things; the simpler one held; the one that looked better in-sample didn’t. A reader who hears it can calibrate everything else you say.