Monitoring with Metrics

Learning Objectives

By the end of this section, you will be able to:

1. Construct and interpret a confusion matrix and explain why it contains more information than accuracy alone
2. Compute precision, recall, and F1 score from scratch and explain which metric matters most for a given application
3. Explain why the decision threshold is a critical hyperparameter and how to tune it for maximum F1
4. Read and plot learning curves to diagnose underfitting, overfitting, and data hunger
5. Use scikit-learn's classification_report and PyTorch-native tensor operations for production metric computation

Where We Left Off

In Section 3, we built the train/validation/test framework: we split data, tracked validation loss, and implemented early stopping. But we monitored only one number — the loss. While loss is essential for gradient-based optimization, it does not directly answer the questions practitioners actually care about: “How many positives did we miss?” “How many of our positive predictions are wrong?” “Should I trust this model in production?”

This section introduces the metrics toolkit that answers those questions. Loss tells the optimizer where to go. Metrics tell you whether the model is useful.

Beyond Loss: Why We Need More Metrics

Suppose you are building a model to detect fraudulent credit card transactions. In your dataset, 99.5% of transactions are legitimate and 0.5% are fraudulent. A model that always predicts “legitimate” has:

• Accuracy: 99.5% — looks fantastic
• Recall: 0% — it catches zero fraud
• Business value: zero — the whole point was to catch fraud

This is the accuracy paradox: on imbalanced datasets, accuracy rewards predicting the majority class and ignores the minority class entirely. The model has high accuracy but is completely useless for its intended purpose.

The Core Insight: Accuracy tells you how often the model is right. But on imbalanced data, being right about the majority class is trivial and uninteresting. What you really need to know is: when the model says “positive,” can you trust it? And how many real positives is it missing?

The Confusion Matrix

The confusion matrix is a 2×2 table that breaks down every prediction into one of four categories. It is the foundation from which all classification metrics are derived.

Every sample in the test set falls into exactly one cell. The four counts are exhaustive and mutually exclusive: $\text{TP} + \text{FP} + \text{TN} + \text{FN} = N$.

The names use a simple convention: the first word (“True” / “False”) says whether the model was correct. The second word (“Positive” / “Negative”) says what the model predicted.

• TP: Model said positive. It was positive. Correct ✓
• FP: Model said positive. It was negative. Wrong ✗ (false alarm)
• TN: Model said negative. It was negative. Correct ✓
• FN: Model said negative. It was positive. Wrong ✗ (missed detection)

Precision, Recall, and F1

Accuracy

The fraction of all predictions that are correct:

$\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{FP} + \text{TN} + \text{FN}}$

Good as a sanity check on balanced datasets. Misleading on imbalanced datasets.

Precision: “When I say positive, am I right?”

$\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}$

High precision means few false alarms. When the model says “positive,” you can trust it. Precision matters most when false positives are expensive: a spam filter that blocks real emails (FP) annoys users, so you want high precision.

Recall: “Did I catch all the positives?”

$\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}$

High recall means few missed detections. Recall matters most when false negatives are dangerous: a cancer screening tool that misses a tumor (FN) is potentially fatal, so you want high recall even at the cost of some false positives.

F1 Score: Balancing Precision and Recall

$\text{F1} = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} = \frac{2\text{TP}}{2\text{TP} + \text{FP} + \text{FN}}$

The F1 score is the harmonic mean of precision and recall. Unlike the arithmetic mean, the harmonic mean penalizes extreme imbalance: if either precision or recall is near zero, F1 is near zero. Both must be high for F1 to be high.

The second row is the key insight: a model with 90% precision but only 10% recall seems “okay” by arithmetic mean (50%) but is terrible by F1 (18%). The harmonic mean refuses to paper over a failing metric.

The Threshold Decision

A classifier typically outputs a probability $p \in [0, 1]$, not a hard class label. To get a binary prediction, we choose a threshold $\tau$:

$\hat{y} = \begin{cases} 1 & \text{if } p \geq \tau \\ 0 & \text{if } p < \tau \end{cases}$

The default $\tau = 0.5$ is often not optimal. Changing the threshold creates a precision-recall tradeoff:

• Lower threshold ($\tau = 0.3$): More samples are predicted positive. Recall increases (catch more positives) but precision decreases (more false alarms).
• Higher threshold ($\tau = 0.7$): Fewer samples are predicted positive. Precision increases (fewer false alarms) but recall decreases (miss more positives).

The optimal threshold depends on the application. To find the threshold that maximizes F1, sweep thresholds from 0 to 1 and pick the one with the highest F1 score. This should be done on the validation set, never on the test set.

ROC and PR Curves

The threshold slider gave us one operating point at a time. A ROC curveand a PR curve sweep all thresholds at once and plot the tradeoff as a single line. Instead of picking one threshold and reporting precision / recall for it, we report the area under the curve — a summary of how well the model ranks positives above negatives, independent of any particular threshold.

ROC-AUC has a clean probabilistic interpretation: if you pick one random positive and one random negative, ROC-AUC is the probability that the model assigns the positive the higher score (Fawcett 2006). An AUC of 0.8125 means the ranking is correct about 81% of the time.

PR-AUC reports a different thing: how precise the model is at every recall level. When the positive class is rare, ROC-AUC can stay deceptively high because the denominator of FPR (the huge pool of negatives) makes even hundreds of false positives look like a tiny FPR. PR-AUC is dominated by precision, so it collapses as soon as false positives start accumulating. Saito & Rehmsmeier (2015) showed this formally; the short version: on heavily imbalanced data, prefer PR-AUC over ROC-AUC.

1import numpy as np
2
3# 8 predictions: model score (probability of class 1) and true label
4scores = np.array([0.95, 0.80, 0.75, 0.60, 0.50, 0.40, 0.30, 0.10])
5y_true = np.array([   1,    1,    0,    1,    0,    1,    0,    0])
6
7P = int((y_true == 1).sum())   # positives  = 4
8N = int((y_true == 0).sum())   # negatives  = 4
9
10# Sweep thresholds from just above the max score down to just below the min.
11# As tau drops, more samples are predicted positive -> both TPR and FPR rise.
12thresholds = np.sort(np.concatenate([[1.01], scores, [-0.01]]))[::-1]
13
14tpr, fpr, precision, recall = [], [], [], []
15for tau in thresholds:
16    pred = scores >= tau
17    tp = int((pred & (y_true == 1)).sum())
18    fp = int((pred & (y_true == 0)).sum())
19    tpr.append(tp / P)
20    fpr.append(fp / N)
21    precision.append(tp / (tp + fp) if (tp + fp) > 0 else 1.0)
22    recall.append(tp / P)
23
24# ROC-AUC: area under (FPR, TPR). Trapezoidal rule on the swept curve.
25roc_auc = np.trapz(tpr, fpr)
26print(f"ROC-AUC = {roc_auc:.4f}")   # 0.8125

scikit-learn gives you the same result in three function calls:

1import numpy as np
2from sklearn.metrics import (
3    roc_curve, auc,
4    precision_recall_curve, average_precision_score,
5)
6
7scores = np.array([0.95, 0.80, 0.75, 0.60, 0.50, 0.40, 0.30, 0.10])
8y_true = np.array([   1,    1,    0,    1,    0,    1,    0,    0])
9
10# --- ROC ---
11fpr, tpr, roc_th = roc_curve(y_true, scores)
12roc_auc = auc(fpr, tpr)
13print(f"ROC-AUC            = {roc_auc:.4f}")   # 0.8125
14
15# --- PR ---
16prec, rec, pr_th = precision_recall_curve(y_true, scores)
17ap = average_precision_score(y_true, scores)
18print(f"PR-AUC (Avg. Prec) = {ap:.4f}")        # 0.8542

Quick decision rule. For balanced binary problems, report ROC-AUC and accuracy at the best threshold. For imbalanced problems (e.g., fraud, rare-disease, spam), always report PR-AUC / Average Precision alongside ROC-AUC — the gap between the two is itself a diagnostic.

Multiclass Metrics

Everything so far assumed two classes. For a $K$-class problem the confusion matrix grows to $K \times K$, with rows indexing the true class and columns indexing the predicted class. The diagonal counts correct predictions; every off-diagonal cell $C_{i,j}$ counts the samples of true class $i$ that were misclassified as class $j$.

Per-class precision and recall come from a one-vs-rest reduction: treat class $k$ as “positive” and every other class as “negative”. Then $\text{precision}_k = C_{k,k} / \sum_i C_{i,k}$ (diagonal over column sum) and $\text{recall}_k = C_{k,k} / \sum_j C_{k,j}$ (diagonal over row sum).

The interesting question is how to collapse $K$ per-class scores into a single headline number. There are three standard choices (Manning, Raghavan & Schütze 2008):

[[3 1 1]
 [1 2 0]
 [1 1 2]]

1import numpy as np
2
3# 3-class problem, 12 samples. y_true is the truth, y_pred is argmax(model output).
4y_true = np.array([0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2])
5y_pred = np.array([0, 0, 0, 1, 2, 1, 1, 0, 2, 2, 1, 0])
6
7K = 3
8conf = np.zeros((K, K), dtype=int)
9for t, p in zip(y_true, y_pred):
10    conf[t, p] += 1
11print("Confusion (rows = true, cols = pred):")
12print(conf)
13
14# Per-class precision / recall / F1
15per_class = {}
16for k in range(K):
17    tp = conf[k, k]
18    fp = conf[:, k].sum() - tp
19    fn = conf[k, :].sum() - tp
20    prec = tp / (tp + fp) if (tp + fp) else 0.0
21    rec  = tp / (tp + fn) if (tp + fn) else 0.0
22    f1   = 2 * prec * rec / (prec + rec) if (prec + rec) else 0.0
23    support = int(conf[k, :].sum())
24    per_class[k] = (prec, rec, f1, support)
25    print(f"class {k}: P={prec:.3f}  R={rec:.3f}  F1={f1:.3f}  support={support}")
26
27# Macro: unweighted mean of per-class F1
28macro_f1 = np.mean([v[2] for v in per_class.values()])
29
30# Weighted: per-class F1 weighted by support
31total_support = sum(v[3] for v in per_class.values())
32weighted_f1 = sum(v[2] * v[3] for v in per_class.values()) / total_support
33
34# Micro: pool TP / FP / FN across classes, then compute F1.
35# For multiclass this equals accuracy.
36tp_tot = int(np.trace(conf))
37fp_tot = int(conf.sum() - tp_tot)   # every non-diagonal is FP for some class
38fn_tot = fp_tot                       # and simultaneously FN for another
39micro_prec = tp_tot / (tp_tot + fp_tot)
40micro_rec  = tp_tot / (tp_tot + fn_tot)
41micro_f1   = 2 * micro_prec * micro_rec / (micro_prec + micro_rec)
42accuracy   = tp_tot / conf.sum()
43
44print(f"macro   F1 = {macro_f1:.3f}")
45print(f"weighted F1 = {weighted_f1:.3f}")
46print(f"micro   F1 = {micro_f1:.3f}  (= accuracy = {accuracy:.3f})")

scikit-learn's classification_report prints the full per-class table and all three averages in a single call:

1import numpy as np
2from sklearn.metrics import (
3    confusion_matrix,
4    classification_report,
5    f1_score,
6)
7
8y_true = np.array([0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2])
9y_pred = np.array([0, 0, 0, 1, 2, 1, 1, 0, 2, 2, 1, 0])
10
11print(confusion_matrix(y_true, y_pred))
12print()
13print(classification_report(y_true, y_pred, digits=3))
14
15print("f1_score macro   :", f1_score(y_true, y_pred, average="macro"))
16print("f1_score weighted:", f1_score(y_true, y_pred, average="weighted"))
17print("f1_score micro   :", f1_score(y_true, y_pred, average="micro"))

Warning on imbalance. Swap the supports to something like 90/5/5 and train a model that only predicts class 0. Accuracy is 90%. Weighted F1 is ≈ 0.85. Macro F1 drops toward $1/3$ because classes 1 and 2 have F1 ≈ 0. If you only reported accuracy you would never notice the failure. This is exactly why macro averaging is the standard metric for imbalanced multiclass problems.

Interactive: Metrics Dashboard

The dashboard below shows 20 binary classification predictions. Drag the threshold slider and watch how precision, recall, F1, and the confusion matrix change in real-time. The precision-recall curve shows the tradeoff as a curve — the optimal operating point (best F1) is marked in green.

Key observations as you experiment:

• Threshold 0.05–0.20: Nearly all samples predicted positive. Recall is 100% (all positives caught) but precision drops because many negatives are incorrectly labeled positive
• Threshold 0.30: The sweet spot for this dataset. F1 peaks at 92.3% — all 12 positives are caught (recall=100%) with only 2 false positives (precision=85.7%)
• Threshold 0.50 (default): 3 positives are missed (recall drops to 75%). F1 = 78.3%. The default is not optimal
• Threshold 0.70+: Zero false positives (precision=100%) but the model misses more and more positives. Recall and F1 drop sharply
• Threshold 0.90+: Only the most confident predictions survive. The model is extremely conservative — perfect precision but catches only 25% of positives

Computing Metrics from Scratch

The code below computes all four confusion matrix entries, derives precision/recall/F1, and sweeps thresholds to find the optimal F1 — all in NumPy. Understanding this from-scratch implementation makes the formulas concrete.

              Predicted
            Pos    Neg
Actual Pos   9(TP)  3(FN)
Actual Neg   2(FP)  6(TN)

1import numpy as np
2
3# ── 20 test samples: ground truth and model probabilities ──
4y_true = np.array([1,0,1,1,0,1,0,1,1,0,1,1,0,1,0,1,1,0,1,0])
5probs  = np.array([0.92,0.15,0.88,0.42,0.22,0.78,0.61,
6                    0.85,0.95,0.18,0.38,0.73,0.12,0.81,
7                    0.55,0.90,0.35,0.08,0.87,0.20])
8
9# ── Step 1: Apply decision threshold ──
10threshold = 0.5
11y_pred = (probs >= threshold).astype(int)
12print(f"Predictions: {y_pred.tolist()}")
13
14# ── Step 2: Build confusion matrix ──
15TP = np.sum((y_true == 1) & (y_pred == 1))
16FP = np.sum((y_true == 0) & (y_pred == 1))
17TN = np.sum((y_true == 0) & (y_pred == 0))
18FN = np.sum((y_true == 1) & (y_pred == 0))
19print(f"TP={TP}, FP={FP}, TN={TN}, FN={FN}")
20
21# ── Step 3: Compute metrics ──
22accuracy = (TP + TN) / len(y_true)
23precision = TP / (TP + FP)
24recall = TP / (TP + FN)
25f1 = 2 * precision * recall / (precision + recall)
26print(f"Accuracy:  {accuracy:.4f}")
27print(f"Precision: {precision:.4f}")
28print(f"Recall:    {recall:.4f}")
29print(f"F1 Score:  {f1:.4f}")
30
31# ── Step 4: Threshold sweep — find best F1 ──
32best_f1 = 0
33best_th = 0
34for th in np.arange(0.05, 1.0, 0.05):
35    yp = (probs >= th).astype(int)
36    tp = np.sum((y_true == 1) & (yp == 1))
37    fp = np.sum((y_true == 0) & (yp == 1))
38    fn = np.sum((y_true == 1) & (yp == 0))
39    pr = tp / (tp + fp) if (tp + fp) > 0 else 0
40    re = tp / (tp + fn) if (tp + fn) > 0 else 0
41    f = 2*pr*re/(pr+re) if (pr + re) > 0 else 0
42    if f > best_f1:
43        best_f1 = f
44        best_th = th
45
46print(f"\nBest F1: {best_f1:.4f} at threshold {best_th:.2f}")

The threshold sweep reveals that the default threshold of 0.5 gives F1 = 0.783, but shifting to 0.30 pushes F1 to 0.923 — a dramatic improvement from changing a single number. The three missed positives (samples 3, 10, 16) all had probabilities between 0.35 and 0.42, just below 0.5 but above 0.3. Lowering the threshold rescues them.

Practical Lesson: Never accept the default threshold of 0.5 without checking. A 5-line threshold sweep on the validation set can improve your F1 by 10+ percentage points. This is one of the highest-ROI optimizations in applied machine learning.

PyTorch and scikit-learn

In production, you use scikit-learn for metrics computation. It handles edge cases (zero division, multiclass averaging, weighted metrics) that our manual code does not. The PyTorch code below shows both the sklearn approach and a pure-PyTorch fallback for environments where sklearn is not available.

[[6, 2],    ← actual=0: TN=6, FP=2
 [3, 9]]    ← actual=1: FN=3, TP=9

1import torch
2import torch.nn as nn
3from sklearn.metrics import (
4    accuracy_score, precision_score, recall_score,
5    f1_score, confusion_matrix, classification_report
6)
7
8# ── Same data as NumPy version ──
9y_true = torch.tensor([1,0,1,1,0,1,0,1,1,0,1,1,0,1,0,1,1,0,1,0])
10probs  = torch.tensor([0.92,0.15,0.88,0.42,0.22,0.78,0.61,
11                        0.85,0.95,0.18,0.38,0.73,0.12,0.81,
12                        0.55,0.90,0.35,0.08,0.87,0.20])
13
14# ── Step 1: Predictions at threshold 0.5 ──
15y_pred = (probs >= 0.5).long()
16
17# ── Step 2: Confusion matrix (sklearn) ──
18cm = confusion_matrix(y_true.numpy(), y_pred.numpy())
19print(f"Confusion matrix:\n{cm}")
20
21# ── Step 3: All metrics in one call ──
22report = classification_report(
23    y_true.numpy(), y_pred.numpy(),
24    target_names=["Negative", "Positive"],
25    digits=4
26)
27print(report)
28
29# ── Step 4: Individual metrics ──
30acc = accuracy_score(y_true.numpy(), y_pred.numpy())
31prec = precision_score(y_true.numpy(), y_pred.numpy())
32rec = recall_score(y_true.numpy(), y_pred.numpy())
33f1 = f1_score(y_true.numpy(), y_pred.numpy())
34print(f"Accuracy:  {acc:.4f}")
35print(f"Precision: {prec:.4f}")
36print(f"Recall:    {rec:.4f}")
37print(f"F1:        {f1:.4f}")
38
39# ── Step 5: PyTorch-native computation (no sklearn) ──
40tp = ((y_true == 1) & (y_pred == 1)).sum().item()
41fp = ((y_true == 0) & (y_pred == 1)).sum().item()
42fn = ((y_true == 1) & (y_pred == 0)).sum().item()
43precision_pt = tp / (tp + fp)
44recall_pt = tp / (tp + fn)
45f1_pt = 2 * precision_pt * recall_pt / (precision_pt + recall_pt)
46print(f"\nPyTorch-native F1: {f1_pt:.4f}")

All three implementations — NumPy manual, sklearn, and PyTorch-native — produce identical results. The sklearn version is the most concise and handles edge cases. The PyTorch-native version is useful when you need metrics computed on GPU tensors without converting to NumPy.

Learning Curves

A learning curve plots a metric (loss or accuracy) against the training epoch (or number of training samples). Comparing the training curve to the validation curve reveals three diagnostic patterns:

How to Read Learning Curves

The gap between the training and validation curves is the generalization gap from Section 3. If the gap is small throughout training, the model generalizes well. If it starts small and grows, the model is beginning to overfit. If it is large from the start, the model is memorizing from the very first epoch (usually means the model is too complex for the available data).

A healthy training run shows both curves decreasing, with the validation curve slightly above the training curve. When the validation curve starts to rise while the training curve continues to fall, it is time to stop (this is exactly what early stopping detects automatically).

Connection to Modern Systems

LLM Evaluation Metrics

Large language models use specialized metrics beyond classification: perplexity $\text{PPL} = \exp(-\frac{1}{T}\sum_t \log P(x_t|x_{<t}))$ measures how well the model predicts the next token. BLEU and ROUGE evaluate text generation quality against references. Human evaluation (Elo ratings, preference rankings) captures qualities that automated metrics miss. The LMSYS Chatbot Arena uses pairwise human preferences to rank models, producing more reliable comparisons than any single automated metric.

Metric Aggregation at Scale

Production systems compute metrics on millions of predictions per second. Key practices: macro averaging (compute metric per class, then average) gives equal weight to rare classes. Micro averaging (pool all predictions, then compute) weights by class frequency. Weighted averaging (macro with class frequency weights) is a compromise. For imbalanced datasets, macro averaging is often preferred because it prevents the majority class from dominating the score.

Monitoring in Production

After deployment, models need continuous monitoring. Data drift occurs when the input distribution changes (e.g., new types of fraud). Concept drift occurs when the relationship between inputs and outputs changes. Production monitoring systems track metrics over time, set alerts when performance degrades, and trigger retraining automatically. Tools like Weights & Biases, MLflow, and TensorBoard provide dashboards for tracking metrics across experiments and deployments.

Beyond Binary Classification

For multiclass problems with $K$ classes, the confusion matrix becomes$K \times K$, and precision/recall/F1 are computed per-class then averaged. For regression problems, metrics include MSE, MAE, and R². For ranking problems, NDCG and MAP measure the quality of the ordering. Every domain has its own metrics, but the principle is the same: measure what matters for your application, not just what is easy to compute.

Summary

In this final section of Chapter 11, we built the metrics toolkit for evaluating neural networks:

1. Accuracy is not enough. On imbalanced datasets, a model that always predicts the majority class has high accuracy but zero usefulness. Always check precision, recall, and F1
2. Confusion matrix breaks predictions into TP, FP, TN, FN. Every classification metric is derived from these four counts: $\text{Precision} = \text{TP}/(\text{TP}+\text{FP})$, $\text{Recall} = \text{TP}/(\text{TP}+\text{FN})$
3. F1 = harmonic mean of precision and recall. It penalizes imbalance: both precision and recall must be high. Use F1 as the default metric for imbalanced binary problems
4. Threshold tuning is a critical optimization. The default 0.5 is often not the best. Sweep thresholds on the validation set and pick the one that maximizes F1. This can improve F1 by 10+ points
5. Learning curves (train vs val metric over epochs) are the most informative single diagnostic. A growing gap signals overfitting. A high plateau signals underfitting

With this, Chapter 11 is complete. You now have the full training toolkit: data loading and batching (Section 1), the training loop with gradient clipping and LR scheduling (Section 2), validation and early stopping (Section 3), and comprehensive metrics (Section 4). In Chapter 12, we will dive deeper into regularization — techniques like dropout and weight decay that directly combat the overfitting we learned to detect in this chapter.

References

• van Rijsbergen, C. J. (1979). Information Retrieval (2nd ed.), Butterworths. The book that introduced the F-measure (originally called the E-measure).
• Manning, C. D., Raghavan, P. & Schütze, H. (2008). Introduction to Information Retrieval, Cambridge University Press. Chapter 8 is the standard reference for macro, micro, and weighted averaging.
• Powers, D. M. W. (2011). Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness and Correlation. Journal of Machine Learning Technologies 2(1), 37–63 (also arXiv:2010.16061). Rigorous derivation of every binary metric.
• Fawcett, T. (2006). An Introduction to ROC Analysis. Pattern Recognition Letters 27(8), 861–874. Canonical ROC reference; defines AUC and its probabilistic interpretation.
• Davis, J. & Goadrich, M. (2006). The Relationship Between Precision-Recall and ROC Curves. ICML 2006.
• Saito, T. & Rehmsmeier, M. (2015). The Precision-Recall Plot is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLoS ONE 10(3), e0118432. Shows that ROC-AUC stays flattering on imbalanced data while PR-AUC collapses.
• Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. ICML 2017 / arXiv:1706.04599. Motivates using log-loss (negative log-likelihood) as a reported metric.
• Pedregosa, F. et al. (2011). Scikit-learn: Machine Learning in Python. JMLR 12, 2825–2830. The sklearn.metrics module used throughout this section.