Pearson versus Spearman

Pearson correlation measures linear co-movement of the raw values: it is the OLS slope between two standardised variables. Spearman correlation is the Pearson correlation of the ranks of the values. That is the whole definition, and it is why Spearman is bounded in its sensitivity to any single observation: an observation can be at most first or last, however extreme its value.

When they disagree

On data with fat tails, a few extreme rows can dominate Pearson while leaving Spearman untouched. Daily equity returns have 46 rows out of 619,040 with moves over 25%, several of them unadjusted corporate actions rather than real moves. A raw correlation on any day containing one of them is largely a statement about that one row.

import pandas as pd, numpy as np

df = pd.read_csv("all_stocks_5yr.csv")
df["date"] = pd.to_datetime(df["date"])
df = df.rename(columns={"Name": "ticker"}).sort_values(["ticker", "date"]).reset_index(drop=True)
per = df.groupby("ticker").size()
df = df[df["ticker"].isin(per[per >= 1000].index)].copy()
g = df.groupby("ticker")
df["ret"] = g["close"].pct_change()
df["fwd_ret"] = g["ret"].shift(-1)
df["sig1"] = -df["ret"]
df = df.dropna(subset=["ret", "fwd_ret"])
test = df[df["date"] > pd.Timestamp("2016-06-30") + pd.Timedelta(days=7)].copy()

def daily_ic(d, sig, tgt="fwd_ret", method="spearman"):
    return d.groupby("date")[[sig, tgt]].apply(lambda x: x[sig].corr(x[tgt], method=method)).dropna()

def report(ic, label):
    m, s, n = ic.mean(), ic.std(), len(ic)
    print("%-10s mean %+.4f   t = %+.2f   hit = %.2f   days = %d" % (label, m, m / (s / np.sqrt(n)), (ic > 0).mean(), n))

report(daily_ic(test, "sig1", method="pearson"), "Pearson")
report(daily_ic(test, "sig1", method="spearman"), "Spearman")

Pearson    mean +0.0063   t = +0.81   hit = 0.53   days = 399
Spearman   mean +0.0156   t = +1.98   hit = 0.55   days = 399

Same signal, same target, same 399 days. Pearson sees nothing distinguishable from zero; Spearman sees a small consistent effect. The extreme rows are adding noise to the raw correlation that swamps the signal.

One day, by eye

The mechanism is visible on any single date that contains an extreme move.

extreme_days = test.loc[test["ret"].abs() > 0.25, "date"].unique()
day = test[test["date"] == extreme_days[0]]
print(day["date"].iloc[0].date(), "| stocks:", len(day), "| largest |ret| that day: %.2f" % day["ret"].abs().max())
print("Pearson  %+.4f" % day["sig1"].corr(day["fwd_ret"]))
print("Spearman %+.4f" % day["sig1"].corr(day["fwd_ret"], method="spearman"))
trim = day[day["ret"].abs() < 0.25]
print("Pearson, that one row removed:  %+.4f" % trim["sig1"].corr(trim["fwd_ret"]))
print("Spearman, that one row removed: %+.4f" % trim["sig1"].corr(trim["fwd_ret"], method="spearman"))

2016-11-01 | stocks: 483 | largest |ret| that day: 0.34
Pearson  -0.1438
Spearman -0.0429
Pearson, that one row removed:  -0.0884
Spearman, that one row removed: -0.0335

Removing one row out of several hundred moves the Pearson correlation by a large fraction of its value and barely moves the Spearman correlation. On a day like this the Pearson number is mostly a statement about one stock.

Which to use

• Spearman for scoring signals on financial or otherwise heavy-tailed data, and for anything where “monotone relationship” is what was meant.
• Pearson when the relationship is expected to be linear and the tails are known to be tame, or when the slope in units is the quantity of interest (then fit OLS and report the slope, not the correlation).
• Both, side by side, when explaining why ranks were used. The pair of lines above is the explanation.

In a sentence

Spearman is Pearson on the ranks, so it measures whether higher goes with higher rather than how linearly, and one crazy observation cannot decide it.