Logistic regression with a majority-class baseline

When the target is binary, “up or down tomorrow”, “big move or not”, the model of first reach is logistic regression, and the number it has to beat is not zero but the majority-class rate: the accuracy of always predicting the more common class. On daily equity returns about 52% of stock-days are up, so a model at 52% accuracy has learned nothing.

Setup and the baseline

import pandas as pd, numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss

df = pd.read_csv("all_stocks_5yr.csv")
df["date"] = pd.to_datetime(df["date"])
df = df.rename(columns={"Name": "ticker"}).sort_values(["ticker", "date"]).reset_index(drop=True)
per = df.groupby("ticker").size()
df = df[df["ticker"].isin(per[per >= 1000].index)].copy()
g = df.groupby("ticker")
df["ret"] = g["close"].pct_change()
df["fwd_ret"] = g["ret"].shift(-1)
df["vol20"] = g["ret"].transform(lambda x: x.rolling(20).std())
df["relvol"] = df["volume"] / g["volume"].transform(lambda x: x.rolling(20).mean())
df["move"] = df["ret"].abs() / df["vol20"]
df = df.replace([np.inf, -np.inf], np.nan).dropna(subset=["fwd_ret", "vol20", "relvol", "move"])

# target 1: direction. target 2: "big move", |tomorrow| more than 1.5x the stock's own vol
df["up"] = (df["fwd_ret"] > 0).astype(int)
df["big"] = (df["fwd_ret"].abs() / df["vol20"] > 1.5).astype(int)

cut, embargo = pd.Timestamp("2016-06-30"), pd.Timedelta(days=7)
train = df[df["date"] <= cut]
test = df[df["date"] > cut + embargo]

for tgt in ("up", "big"):
    majority = train[tgt].mean() if train[tgt].mean() > 0.5 else 1 - train[tgt].mean()
    print("%-4s majority-class rate on train %.3f | share of positives on test %.3f" % (tgt, majority, test[tgt].mean()))

up   majority-class rate on train 0.519 | share of positives on test 0.524
big  majority-class rate on train 0.856 | share of positives on test 0.136

The baseline accuracy on the test period is whichever of the positive share or one minus it is larger, using the class that was the majority on train.

Fit, score, compare

Features are ranked within date so that the scale is the same every day, and the model is fitted on the build period only.

feats = ["ret", "relvol", "move"]
for c in feats:
    df[c + "_r"] = df.groupby("date")[c].rank(pct=True) - 0.5
X = [c + "_r" for c in feats]
train = df[df["date"] <= cut]
test = df[df["date"] > cut + embargo]

for tgt in ("up", "big"):
    model = LogisticRegression().fit(train[X].values, train[tgt].values)
    p = model.predict_proba(test[X].values)[:, 1]
    major = int(train[tgt].mean() > 0.5)
    base_acc = (test[tgt].values == major).mean()
    base_p = np.full(len(test), train[tgt].mean())
    acc = ((p > 0.5).astype(int) == test[tgt].values).mean()
    print("%-4s baseline acc %.4f  model acc %.4f | baseline log-loss %.4f  model log-loss %.4f | coef %s"
          % (tgt, base_acc, acc, log_loss(test[tgt].values, base_p), log_loss(test[tgt].values, p), dict(zip(feats, model.coef_[0].round(3)))))

up   baseline acc 0.5238  model acc 0.5241 | baseline log-loss 0.6921  model log-loss 0.6921 | coef {'ret': -0.073, 'relvol': 0.05, 'move': -0.08}
big  baseline acc 0.8636  model acc 0.8636 | baseline log-loss 0.3986  model log-loss 0.3950 | coef {'ret': -0.093, 'relvol': 0.619, 'move': 0.12}

For direction, the model matches the majority-class rate to three decimals and the log-loss is unchanged: nothing learned, which is the honest answer for a one-day-ahead sign. For big moves, accuracy is identical because at a 0.5 threshold the model never predicts the 14% minority class, but the log-loss improves from 0.3986 to 0.3950, so the probabilities carry information the constant rate does not; the coefficient on relative volume (0.62) is five times the others.

Reading it

• Accuracy against the majority-class rate answers “does it get more right than always guessing the common class”. For direction it will barely move; direction is close to unpredictable a day ahead.
• Log-loss against the constant-probability baseline answers “are the probabilities better calibrated than the base rate”. It rewards a model that says 0.6 when it should and punishes overconfidence; it is the better score when the classes are unbalanced or the decision threshold is not 0.5.
• Coefficients on within-date ranks are comparable to each other. A large coefficient on move for the big target is volatility clustering; a coefficient on relvol on top of it is the relative-volume finding in classifier form.

What not to do

• Do not report accuracy without the majority-class rate next to it.
• Do not use a random train/test split on a time-ordered panel. Rows from the same date share the market’s move, and a random split leaks it. See chronological splits with an embargo.
• Do not tune the threshold or the regularisation on the test period. If a threshold matters, choose it on the build period and report it once.