Choosing a model, in order of reach

For a small research question the model is rarely where the value is. The reasoning about the target, the split and the baseline is; a reader is grading the honesty of the number, not the sophistication of what produced it. So reach for models in order of how little they can hide.

The ladder

Rung	Use when	Base-stack implementation	What you report
1. Correlation / rank IC	almost always first	groupby("date").apply(corr)	mean IC, t, hit rate
2. OLS	one or two features, a slope with an error bar wanted	np.linalg.lstsq, or sklearn.linear_model.LinearRegression	slope, t, R²
3. Logistic regression	binary target: up/down, big move/not	sklearn.linear_model.LogisticRegression	accuracy against the majority-class rate; log-loss
4. Ridge	several correlated features	sklearn.linear_model.Ridge	out-of-sample IC of the prediction
5. Tree models	non-financial tabular data with interactions, and time to spare	sklearn.ensemble.RandomForestRegressor	the same, plus feature importances

Not on the ladder for a short study: gradient boosting, hyperparameter searches, anything needing a GPU or a long fit. They add fitting time and remove the ability to say what the model did, and with a hold-out reported once there is no budget for tuning anyway.

Rung 1 is usually enough

The rank IC already answers “does this rank predict that rank” with an error bar, and it makes no distributional assumption. Move up a rung only when the question asks for something a correlation cannot give: a slope in units (“how much return per unit of signal?”), a probability, or a combination of several features.

Rung 2: OLS

The numpy implementation with standard errors is on its own page, OLS from first principles. The sklearn version is one line but gives no standard errors, which is why the numpy version exists.

Rung 3: logistic, with the baseline that matters

A classifier’s accuracy means nothing without the majority-class rate. If 56% of days are up, a model that says “up” every day scores 56%. See logistic regression with a majority-class baseline.

Rung 4: ridge over a few ranked features

When there are several features the honest version is: rank each within date, fit a linear combination on the build period, and score the prediction with the same daily IC used for a single signal. Ridge rather than plain OLS because ranked features are correlated and the penalty keeps the weights sane.

import pandas as pd, numpy as np
from sklearn.linear_model import Ridge

df = pd.read_csv("all_stocks_5yr.csv")
df["date"] = pd.to_datetime(df["date"])
df = df.rename(columns={"Name": "ticker"}).sort_values(["ticker", "date"]).reset_index(drop=True)
per = df.groupby("ticker").size()
df = df[df["ticker"].isin(per[per >= 1000].index)].copy()
g = df.groupby("ticker")
df["ret"] = g["close"].pct_change()
df["fwd_ret"] = g["ret"].shift(-1)
df["rev1"] = -df["ret"]
df["rev5"] = -g["ret"].transform(lambda x: x.rolling(5).sum())
df["relvol"] = df["volume"] / g["volume"].transform(lambda x: x.rolling(20).mean())
df = df.replace([np.inf, -np.inf], np.nan).dropna(subset=["fwd_ret", "rev1", "rev5", "relvol"])

feats = ["rev1", "rev5", "relvol"]
for c in feats:                                            # rank within date, centred at zero
    df[c + "_r"] = df.groupby("date")[c].rank(pct=True) - 0.5
df["y_r"] = df.groupby("date")["fwd_ret"].rank(pct=True) - 0.5

cut, embargo = pd.Timestamp("2016-06-30"), pd.Timedelta(days=7)
train = df[df["date"] <= cut].copy()
test = df[df["date"] > cut + embargo].copy()
X = [c + "_r" for c in feats]

model = Ridge(alpha=1.0).fit(train[X].values, train["y_r"].values)
print("weights:", dict(zip(feats, model.coef_.round(4))))
test["pred"] = model.predict(test[X].values)

def daily_ic(d, sig, tgt="fwd_ret"):
    return d.groupby("date")[[sig, tgt]].apply(lambda x: x[sig].corr(x[tgt], method="spearman")).dropna()

def report(ic, label):
    m, s, n = ic.mean(), ic.std(), len(ic)
    print("%-22s mean IC %+.4f   t = %+.2f   hit = %.2f   days = %d" % (label, m, m / (s / np.sqrt(n)), (ic > 0).mean(), n))

for c in feats:
    report(daily_ic(test, c), c + " alone")
report(daily_ic(test, "pred"), "ridge combination")

weights: {'rev1': 0.0078, 'rev5': 0.0124, 'relvol': 0.0099}
rev1 alone             mean IC +0.0156   t = +1.98   hit = 0.55   days = 399
rev5 alone             mean IC +0.0073   t = +0.93   hit = 0.51   days = 399
relvol alone           mean IC -0.0020   t = -0.50   hit = 0.49   days = 399
ridge combination      mean IC +0.0110   t = +1.52   hit = 0.53   days = 399

The combination is worse than the best single feature. On the build period the fit gave the five-day reversal and relative volume weights comparable to the one-day reversal’s, and out of sample those two carry nothing for this target (relative volume predicts the size of tomorrow’s move, not its sign), so they dilute the one feature that works. That is the usual outcome of rung 4 on a short study, and it is a legitimate finding: a linear combination of the three did not beat the simplest one. It is also selection bias from the inside in model form: the fit could not tell noise from signal on one period.

The weights were fitted on the build period only. Scoring the prediction with the daily IC keeps the model on the same footing as a single signal, so the comparison is fair.

Rung 5, briefly

On a non-financial table with real interactions (trip distance × time of day, say), a random forest with default settings and a chronological split is a reasonable second model after OLS. Report it the same way as the linear one, on the same hold-out, and show the feature importances only as a hint of what to look at next. On daily equity returns it will overfit noise and there is no time to show otherwise.