t-statistics from repeated looks

Almost every number on this site with an error bar is built the same way: measure something once per period, then treat the per-period values as a sample.

where $s$ is the sample standard deviation of the $x_i$. For a daily rank IC, $x_i$ is the IC on day $i$ and $n$ is the number of days. A $|t|$ above about 2 is the usual threshold for “distinguishable from zero”, with the caveats below.

import pandas as pd, numpy as np

def summary(x):
    x = pd.Series(x).dropna()
    m, se = x.mean(), x.std() / np.sqrt(len(x))
    return {"mean": round(m, 4), "se": round(se, 4), "t": round(m / se, 2), "n": len(x)}

What it assumes

That the $x_i$ are roughly independent draws with a common mean. For per-day cross-sectional correlations against a one-day-ahead target, that is reasonable: day $i$‘s IC uses day $i$‘s signals and day $i+1$‘s outcomes, and day $i+1$‘s IC uses different outcomes.

Three common ways the assumption fails:

1. Overlapping targets. If the target is the next five days’ return, day $i$ and day $i+1$ share four of their five outcome days. Consecutive ICs are strongly correlated and $n$ overstates the number of independent looks.
2. A single series. Predicting an aggregate (the market’s return tomorrow from today’s) gives one number per day, but those numbers are all one time series: the “repeats” have to be blocks of time. See single series or panel.
3. Regime dependence. If the mean itself drifts (the effect exists in one year and not the next), the t-statistic on the pooled sample answers a question about the average that may not be the question asked. Show the IC by year alongside it.

The cheap correction for overlap

With a horizon of $h$ days, adjacent ICs overlap on $h-1$ days. A conservative correction treats the effective sample size as $n/h$, which multiplies the standard error by $\sqrt{h}$ and divides the t-statistic by the same:

df = pd.read_csv("all_stocks_5yr.csv")
df["date"] = pd.to_datetime(df["date"])
df = df.rename(columns={"Name": "ticker"}).sort_values(["ticker", "date"]).reset_index(drop=True)
per = df.groupby("ticker").size()
df = df[df["ticker"].isin(per[per >= 1000].index)].copy()
g = df.groupby("ticker")
df["ret"] = g["close"].pct_change()
df["sig1"] = -df["ret"]
test_start = pd.Timestamp("2016-06-30") + pd.Timedelta(days=7)

def daily_ic(d, sig, tgt):
    return d.groupby("date")[[sig, tgt]].apply(lambda x: x[sig].corr(x[tgt], method="spearman")).dropna()

for h in (1, 5, 20):
    col = "fwd%d" % h
    df[col] = g["ret"].transform(lambda x: x.shift(-h).rolling(h).sum())    # return over days t+1 .. t+h
    d = df[df["date"] > test_start].dropna(subset=["sig1", col])
    ic = daily_ic(d, "sig1", col)
    m, s, n = ic.mean(), ic.std(), len(ic)
    t_naive = m / (s / np.sqrt(n))
    print("h = %2d   mean IC %+.4f   naive t %+.2f   overlap-adjusted t %+.2f   days %d" % (h, m, t_naive, t_naive / np.sqrt(h), n))

h =  1   mean IC +0.0156   naive t +1.98   overlap-adjusted t +1.98   days 399
h =  5   mean IC +0.0031   naive t +0.42   overlap-adjusted t +0.19   days 395
h = 20   mean IC +0.0031   naive t +0.43   overlap-adjusted t +0.10   days 380

The naive t at a 20-day horizon is computed on 380 “observations” of which only about 19 are independent. Reporting the naive number would be a mistake even when, as here, it is not significant either way.

A less conservative alternative is a Newey–West standard error with $h-1$ lags. It needs a few more lines and gives numbers between the naive and the $\sqrt h$-adjusted versions; the adjustment above is the one to use when there is no time to argue about bandwidths.

Hit rate as a second summary

The share of periods with $x_i > 0$ is a robustness check on the mean. A mean IC carried by a few very good days will have a modest hit rate (near 0.5); a small consistent effect has a hit rate visibly above it. Under the null of no effect the hit rate has standard error $0.5/\sqrt{n}$, so with 400 days anything above about 0.55 is itself unlikely by chance.

ic1 = daily_ic(df[df["date"] > test_start].dropna(subset=["sig1", "fwd1"]), "sig1", "fwd1")
print("hit rate %.3f   se under null %.3f" % ((ic1 > 0).mean(), 0.5 / np.sqrt(len(ic1))))

hit rate 0.546   se under null 0.025

Reporting

Report mean, t and n together, say what one repeat is, and if the repeats overlap say by how much and which correction was used. “IC 0.015, t 2.0 over 399 daily looks at a one-day target” is a complete sentence; “t = 2” on its own is not.