Extreme values: real events or artefacts?
Forty-six daily moves over 25% in a file of 619,040 rows. How to tell a bad print from a split from a real event by looking at two of them, and the three policies available once you have decided.
Every financial file has a tail of moves that are too large to be ordinary. Some are real: a drug trial, an earnings shock, a takeover. Some are artefacts: an unadjusted split, a spin-off, a mispunched price. There is never time to check every one, but there is always time to check two, and two is enough to choose a policy.
Find them
import pandas as pd, numpy as np
df = pd.read_csv("all_stocks_5yr.csv")
df["date"] = pd.to_datetime(df["date"])
df = df.rename(columns={"Name": "ticker"}).sort_values(["ticker", "date"]).reset_index(drop=True)
g = df.groupby("ticker")
df["ret"] = g["close"].pct_change()
df["prev_close"] = g["close"].shift(1)
df["next_ret"] = g["ret"].shift(-1)
big = df[df["ret"].abs() > 0.25].sort_values("ret")
print(len(big), "rows with |return| > 25%")
print(big[["date", "ticker", "prev_close", "close", "ret", "next_ret", "volume"]].head(6).to_string(index=False))
print(big[["date", "ticker", "prev_close", "close", "ret", "next_ret", "volume"]].tail(4).to_string(index=False))46 rows with |return| > 25%
date ticker prev_close close ret next_ret volume
2015-07-02 NI 45.45 16.990 -0.626183 -0.008829 16010299
2015-07-20 EBAY 66.29 28.570 -0.569015 0.001050 37485739
2014-08-07 DISCK 81.22 39.780 -0.510219 0.029663 2670657
2014-08-07 DISCA 83.02 40.975 -0.506444 0.031849 3850670
2016-05-19 LNT 35.35 17.870 -0.494484 1.009513 1231722
2015-07-01 BAX 69.93 38.860 -0.444301 -0.014411 24435196
date ticker prev_close close ret next_ret volume
2014-06-24 VRTX 66.61 93.53 0.404144 -0.039239 25236837
2016-04-22 AMD 2.62 3.99 0.522901 -0.135338 143265305
2013-04-19 VRTX 52.87 85.60 0.619066 -0.038201 24894974
2016-05-20 LNT 17.87 35.91 1.009513 -0.004456 997954Three signatures
Reading the columns above answers most cases without leaving the table.
| Signature | What it is | Example above |
|---|---|---|
| A large move that reverses almost exactly the next day | a bad print: one wrong close | −49% then +101% the next day on the same ticker |
| A permanent level shift with a round ratio (½, ⅓, 1/7) and an ordinary next day | an unadjusted split or stock dividend | close almost exactly halves on two share classes of one company, same day |
| A permanent level shift with a non-round ratio and a volume spike | a spin-off (the parent gives up part of its value), or a real event | close falls to 37% of the previous day, next day flat, volume four times the days before |
A real earnings or trial move usually has the third signature with news attached; a spin-off has it without. Both are “real” in the sense that a holder experienced them, but a spin-off holder also received shares in the new company, so the close-to-close return overstates the loss.
The ratio test
For splits, the ratio of the new close to the previous close clusters at simple fractions.
big = big.assign(ratio=(big["close"] / big["prev_close"]).round(3))
print(big.loc[big["ret"] < 0, ["ticker", "date", "ratio", "next_ret"]].sort_values("ratio").head(8).to_string(index=False))ticker date ratio next_ret
NI 2015-07-02 0.374 -0.008829
EBAY 2015-07-20 0.431 0.001050
DISCK 2014-08-07 0.490 0.029663
DISCA 2014-08-07 0.494 0.031849
LNT 2016-05-19 0.506 1.009513
BAX 2015-07-01 0.556 -0.014411
BHGE 2017-07-05 0.646 -0.006443
WMB 2016-02-08 0.652 0.073477Values near 0.5 with an ordinary next day are two-for-one splits or stock dividends. Values that reverse next day are prints. Everything else needs a second look or gets the default policy.
Two by eye
Print a window of a few days around the two most extreme rows and read them.
for _, row in pd.concat([big.head(1), big.tail(1)]).iterrows():
tk, dt = row["ticker"], row["date"]
w = df[(df["ticker"] == tk) & (df["date"].between(dt - pd.Timedelta(days=6), dt + pd.Timedelta(days=6)))]
print("\n%s around %s" % (tk, dt.date()))
print(w[["date", "close", "volume", "ret"]].to_string(index=False))NI around 2015-07-02
date close volume ret
2015-06-26 46.25 2947353 -0.006231
2015-06-29 45.95 2179037 -0.006486
2015-06-30 45.59 2910080 -0.007835
2015-07-01 45.45 3740554 -0.003071
2015-07-02 16.99 16010299 -0.626183
2015-07-06 16.84 11162984 -0.008829
2015-07-07 17.12 10865443 0.016627
2015-07-08 17.14 7037833 0.001168
LNT around 2016-05-20
date close volume ret
2016-05-16 36.985 806028 0.002711
2016-05-17 35.990 1356858 -0.026903
2016-05-18 35.350 1061648 -0.017783
2016-05-19 17.870 1231722 -0.494484
2016-05-20 35.910 997954 1.009513
2016-05-23 35.750 1090320 -0.004456
2016-05-24 36.390 1143051 0.017902
2016-05-25 36.430 1705007 0.001099
2016-05-26 36.730 1300423 0.008235The first is a permanent shift to 37% of the previous close with volume four to five times the days before and no rebound: an unadjusted corporate action (the company separated a large business that month; a holder received shares in the new entity, so the close-to-close return overstates the loss). The second is one wrong close, down 49% and back up 101% the next day, with ordinary volume throughout: a bad print. Neither is a return anyone experienced.
Three policies
- Drop the rows. Simple, honest if said, but loses the real events along with the artefacts, and a “buy the losers” study loses precisely the rows where it might matter.
- Winsorise. Clip returns to, say, ±20%. Keeps the rows, caps their influence, changes the numbers in a way that needs a footnote.
- Use ranks. Score with Spearman correlation and decile buckets, so that the largest move counts as “largest” and no more. Keeps everything, changes nothing else, and is the policy used throughout this site. See Pearson vs Spearman.
Whichever policy is chosen, the sensitivity check is to run the final number with the extreme rows removed and show it did not move. For the reversal signal on the hold-out it did not: IC +0.0156 with them, +0.0156 without. See attacking your own result.
What to write down
One line: “46 rows with |return| > 25%; the two largest inspected: one bad print, one unadjusted corporate action; kept, neutralised by ranks; result unchanged without them.” That line goes on the data slide.