Groupby: transform versus apply versus agg

groupby splits a frame into groups. What happens next depends on which of three verbs follows, and choosing the wrong one produces either an alignment error or, worse, a silently misaligned column.

import pandas as pd, numpy as np

df = pd.read_csv("all_stocks_5yr.csv")
df["date"] = pd.to_datetime(df["date"])
df = df.rename(columns={"Name": "ticker"}).sort_values(["ticker", "date"]).reset_index(drop=True)
df["ret"] = df.groupby("ticker")["close"].pct_change()
g = df.groupby("ticker")

agg: one value per group

The result is as long as the number of groups. Use it for summaries you will look at or merge back explicitly.

s = g["ret"].agg(["mean", "std", "count"])
print(s.shape)
print(s.head(3).round(5).to_string())

transform: one value per row, aligned to the original index

The function receives one group’s Series and must return either a scalar (broadcast to the group) or a Series of the same length. The result has the same index as df, so it can be assigned straight into a new column.

df["ret_mean_own"] = g["ret"].transform("mean")                          # scalar broadcast
df["vol20"] = g["ret"].transform(lambda x: x.rolling(20).std())          # same-length Series
df["ret_z_own"] = (df["ret"] - df["ret_mean_own"]) / g["ret"].transform("std")
print(df[["ticker", "date", "ret", "vol20"]].dropna().head(2).to_string(index=False))

This is the verb for every per-group rolling window, per-group z-score and per-group normalisation. If the phrase in your head is “compute this per ticker and put it on each row”, it is transform.

apply: whatever the function returns

apply passes each group (a DataFrame if you did not select a column, a Series if you did) and stitches together whatever comes back. It is the most flexible verb and the slowest, and the shape of its output depends on the function.

# scalar per group -> Series indexed by group
first_close = g["close"].apply(lambda x: x.iloc[0])
print(first_close.head(3).to_string())

# Series per group -> stacked (group, key) index; unstack() gives one row per group
stats = g["ret"].apply(lambda x: pd.Series({"n": int(x.notna().sum()), "worst": x.min()})).unstack()
print(stats.head(3).to_string())

# per-date correlation between two columns -> Series indexed by date
dfc = df.dropna(subset=["ret"]).copy()
dfc["fwd_ret"] = dfc.groupby("ticker")["ret"].shift(-1)
dfc = dfc.dropna(subset=["fwd_ret"])
ic = dfc.groupby("date")[["ret", "fwd_ret"]].apply(lambda x: x["ret"].corr(x["fwd_ret"], method="spearman"))
print(ic.head(3).round(4).to_string())

Two habits keep apply safe across pandas versions:

1. Select columns before apply. g[["a", "b"]].apply(f) rather than g.apply(f). The function only sees what it needs, and it sidesteps the pandas 2.2 change in whether grouping columns are passed to f.
2. Do not use apply where transform works. g["ret"].apply(lambda x: x.rolling(20).std()) returns a Series with a two-level (ticker, row) index that will not assign cleanly into df. transform returns the aligned version.

The misalignment trap

df["vol20"] = g["ret"].apply(lambda x: x.rolling(20).std())    # MultiIndex result: assignment fills with NaN or raises
df["vol20"] = g["ret"].transform(lambda x: x.rolling(20).std())  # aligned: this is the one

Speed

For large panels transform with a lambda is a Python loop over groups; a few hundred groups is fine, a few hundred thousand is not. Named aggregations (transform("mean")) and vectorised expressions are much faster when they exist. For a study with 500 groups and 600,000 rows, everything on this page runs in seconds.

Summary

You want	Verb	Result length
one summary number per group	agg	number of groups
a per-group computation on every row	transform	same as the frame
a per-group correlation, regression, or custom object	apply on selected columns	depends on the function