Classification metrics, and which one answers the question

A classifier produces a probability, or at least a score, for every row. Everything after that is a choice: whether to report how well the scores rank the rows, how well they calibrate as probabilities, or how well a decision made by thresholding them turns out. Those are three different questions, each with its own metric and its own baseline. Accuracy answers a fourth question that is rarely the one asked.

One block, every metric

On a marketing table with 11.7% positives, an honest logistic model and the constant-rate baseline. The model is the one from the bank marketing study, without the leaked column.

import pandas as pd, numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, average_precision_score, log_loss, brier_score_loss, precision_score, recall_score

df = pd.read_csv("bank_marketing.csv", sep=";")
df["y"] = (df["y"] == "yes").astype(int)
cat = ["job", "marital", "education", "default", "housing", "loan", "contact", "month", "poutcome"]
num = ["age", "balance", "day", "campaign", "pdays", "previous"]
rng = np.random.RandomState(0)
idx = rng.permutation(len(df))
n = int(0.7 * len(df))
tr, te = df.iloc[idx[:n]], df.iloc[idx[n:]]
Xtr = pd.concat([tr[num].astype(float), pd.get_dummies(tr[cat], dtype=float)], axis=1)
Xte = pd.concat([te[num].astype(float), pd.get_dummies(te[cat], dtype=float)], axis=1).reindex(columns=Xtr.columns, fill_value=0.0)
mu, sd = Xtr.mean(), Xtr.std().replace(0, 1)
p = LogisticRegression(max_iter=2000).fit(((Xtr - mu) / sd).values, tr["y"].values).predict_proba(((Xte - mu) / sd).values)[:, 1]
y = te["y"].values
base = np.full(len(y), tr["y"].mean())                                   # the constant-probability baseline

print("positives in test: %.3f" % y.mean())
print("%-22s %8s %8s" % ("", "model", "baseline"))
print("%-22s %8.3f %8.3f   <- majority class scores this without a model" % ("accuracy at 0.5", ((p > 0.5) == y).mean(), ((base > 0.5) == y).mean()))
print("%-22s %8.3f %8.3f   <- 0.5 is a coin toss" % ("ROC-AUC", roc_auc_score(y, p), roc_auc_score(y, base)))
print("%-22s %8.3f %8.3f   <- the positive rate is the floor" % ("PR-AUC", average_precision_score(y, p), average_precision_score(y, base)))
print("%-22s %8.3f %8.3f   <- lower is better; the base rate's log-loss is the ceiling" % ("log-loss", log_loss(y, p), log_loss(y, base)))
print("%-22s %8.3f %8.3f   <- lower is better; p(1-p) of the base rate" % ("Brier score", brier_score_loss(y, p), brier_score_loss(y, base)))

positives in test: 0.119
                          model baseline
accuracy at 0.5           0.891    0.881   <- majority class scores this without a model
ROC-AUC                   0.755    0.500   <- 0.5 is a coin toss
PR-AUC                    0.405    0.119   <- the positive rate is the floor
log-loss                  0.308    0.366   <- lower is better; the base rate's log-loss is the ceiling
Brier score               0.087    0.105   <- lower is better; p(1-p) of the base rate

What each one measures

Accuracy at a 0.5 threshold. The share of rows classified correctly. With 12% positives the constant “no” scores 0.88 and the model 0.89: accuracy is measuring the base rate, and the model’s contribution is the third decimal. Report it only next to the majority-class rate, and never as the headline for an imbalanced target. See logistic regression with a majority-class baseline.

ROC-AUC. The probability that a randomly chosen positive is scored above a randomly chosen negative. It measures ranking and is indifferent to the base rate and to calibration: multiply every probability by ten and it does not change. Baseline 0.5. It is the right headline when the model’s job is to order rows (whom to call first) and the number of rows acted on is not fixed.

PR-AUC (average precision). Precision averaged over recall levels. Also a ranking metric, but it weights the top of the ranking and the rare class, so it moves more than ROC-AUC when the positives are rare and the question is “how clean is the top of the list”. Its floor is the positive rate, not 0.5, which is why a PR-AUC of 0.40 on a 12% target is respectable and the same number on a 40% target is nothing.

Log-loss (cross-entropy). The mean of −log p for the true class. It scores the probabilities themselves: a confident wrong answer is punished hard, and a model that says 0.12 for everyone scores exactly the base rate’s entropy. It is the metric to optimise when the probabilities will be used as probabilities: expected values, capacity planning, anything that multiplies p by a cost.

Brier score. Mean squared error of the probability against the 0/1 outcome. Same purpose as log-loss, gentler on confident mistakes, easier to explain. Baseline p(1 − p) at the base rate.

Calibration. Not a single number: a table of predicted versus observed rate by decile of the prediction. A model can rank perfectly and be miscalibrated (every probability too low by a factor of two), which is what happens when the base rate moves after training; see bank marketing, section 5 and flight delays, section 5.

dec = pd.qcut(p, 10, labels=False, duplicates="drop")
cal = pd.DataFrame({"p": p, "y": y}).groupby(dec).agg(mean_predicted=("p", "mean"), observed_rate=("y", "mean"))
print(cal.round(3).T.to_string())

                    0      1      2      3      4      5      6      7      8      9
mean_predicted  0.018  0.029  0.046  0.061  0.074  0.088  0.105  0.125  0.168  0.453
observed_rate   0.032  0.044  0.060  0.063  0.066  0.081  0.086  0.111  0.162  0.489

The threshold is a decision, not a metric

Precision and recall exist only once a threshold has been chosen, and the threshold is chosen by whoever owns the costs. The model’s job is the curve.

print("threshold  flagged  precision  recall")
for t in (0.5, 0.3, 0.2, 0.1):
    flag = p > t
    print("   %.1f     %5.1f%%    %.3f     %.3f" % (t, 100 * flag.mean(), precision_score(y, flag), recall_score(y, flag)))

threshold  flagged  precision  recall
   0.5       3.3%    0.649     0.180
   0.3       7.2%    0.544     0.330
   0.2      11.1%    0.465     0.431
   0.1      37.7%    0.220     0.694

At 0.5 the model flags 3% of clients and finds 18% of the subscribers; at 0.1 it flags 38% and finds 69%. Neither is “the” performance. If a call costs $c$ and a subscription is worth $v$, a flag pays when precision exceeds $c / v$, and the threshold follows from that ratio. Report the table or the curve; let the decision-maker pick the point. The hotel-cancellation study works one such decision through with explicit assumed costs.

Which to report

The question	Headline	Baseline	Also show
does the model order rows usefully?	ROC-AUC	0.5	PR-AUC if positives are rare
how good is the top of the list?	PR-AUC, precision at the volume that will be acted on	positive rate	the precision/recall table
are the probabilities right?	log-loss or Brier, plus a calibration table	the constant base rate	calibration by period if the base rate can drift
what happens if we act at this threshold?	precision, recall, share flagged	majority class	the cost assumptions that chose the threshold
is the model better than nothing?	any of the above against its baseline		never accuracy alone

Class weights and resampling change the threshold the model effectively uses and the calibration of its probabilities; they do not change what the ranking metrics measure. If the positives are rare, start by reporting the right metric at the natural threshold before reaching for either.