Why Safety Matters: NASA Score vs. RMSE

Two Doctors, Same RMSE, Different Outcomes

Imagine two diagnosticians estimating how many days a patient has left before a cardiac event. Both are wrong by the same average margin — say, six days. Doctor A's mistakes lean early: she usually says “next week” when the real event is two weeks out. Doctor B's mistakes lean late: he typically says “two weeks” when the patient codes in seven days. On any symmetric error metric the two doctors look identical. In the ICU they are not remotely the same person.

That story is the entire reason this section exists. The dominant accuracy metric in regression is RMSE, and RMSE cannot tell A from B. The maintenance community noticed that almost twenty years ago — the result is the NASA scoring function, an exponential, asymmetric cost that bakes into the metric the asymmetry that exists in reality.

RMSE: The Comfortable but Symmetric Metric

RMSE is what every regression paper reports because it has every property a metric should have: same units as the target, differentiable everywhere, cheap to compute, and well-understood. Given predictions $\hat{y}_i$ and ground-truth labels $y_i$ over a test set of size $N$,

$\text{RMSE} \;=\; \sqrt{\,\frac{1}{N}\sum_{i=1}^{N}(\hat{y}_i - y_i)^2\,}.$

The single property that ruins RMSE for safety-critical RUL is the square. Squaring $(\hat{y}_i - y_i)$ erases the sign of the error: an error of $+10$ cycles and an error of $-10$ cycles contribute identically.

The NASA Score: Asymmetry Made Quantitative

Saxena, Goebel and colleagues introduced the asymmetric scoring function alongside the original C-MAPSS dataset in 2008. For a single test sample, with $d_i = \hat{y}_i - y_i$,

$s_i \;=\; \begin{cases} e^{-d_i / 13} - 1 & d_i < 0 \quad\text{(early prediction)}\\[4pt] e^{d_i / 10} - 1 & d_i \ge 0 \quad\text{(late prediction)} \end{cases}$

The total score over the test set is the sum $S = \sum_{i=1}^{N} s_i$. Lower is safer, zero is perfect, and there is no upper bound — one really late prediction can dominate the entire sum.

Two design decisions encode the safety priorities. First, the denominators differ: 13 on the early side, 10 on the late side. Every 13 cycles of earliness multiplies the cost by $e \approx 2.72$; every 10 cycles of lateness does the same. Lateness compounds 30% faster than earliness. Second, the function is exponential, not linear or quadratic — doubling the lateness more than doubles the penalty. A 30-cycle late prediction is worth nineteen 1-cycle late predictions.

The math, decoded

The piecewise formula in the original paper is given in shorthand — no sign condition is written next to it. Make it explicit with $d = \hat{\text{RUL}} - \text{RUL}$:

• $d < 0$ — early prediction: the model says fewer cycles remain than the truth. Conservative. Maintenance is scheduled too soon and life is wasted.
• $d > 0$ — late prediction: the model says more cycles remain than the truth. Dangerous. The engine may be near failure while the model claims plenty of life left.

Worked example: same |d|, two penalties

Take a single engine with true $\text{RUL} = 20$ cycles and consider two predictions that are wrong by the same 20 cycles in opposite directions.

Same absolute error of 20 cycles, but the late penalty is ~75% larger than the early one. RMSE would call these equally bad — both contribute $20^2 = 400$ to the squared error. The NASA score refuses to.

Engineering picture. A model saying “this engine still has 40 cycles left” when it actually has only 20 is dangerous — maintenance arrives too late. A model saying “0 cycles left” when the truth is 20 is conservative and costly, but safer. The asymmetric score encodes that asymmetry of consequences.

Why 10 and 13 — the role of the denominators

Rewrite both branches in terms of $|d|$:

$s_{\text{early}}(d) \;=\; e^{\,|d|/13} - 1, \qquad s_{\text{late}}(d) \;=\; e^{\,|d|/10} - 1.$

For any non-zero error, the late branch divides by the smaller number, which produces a larger exponent — and exponentials magnify any advantage in the exponent very quickly:

$\frac{|d|}{10} \;>\; \frac{|d|}{13} \;\;\Longrightarrow\;\; e^{|d|/10} \;>\; e^{|d|/13}.$

With $|d|=20$: $20/10 = 2$ vs $20/13 \approx 1.54$, so $e^{2}-1 \approx 6.39$ vs $e^{1.54}-1 \approx 3.66$. The denominators are tuning knobs: 13 = softer growth on the safe side, 10 = sharper growth on the dangerous side. The paper does not derive these constants from first principles — they are the convention the C-MAPSS leaderboard has used since 2008 — but their role is precise: make every cycle of lateness 30% more painful than the equivalent cycle of earliness.

Interactive: Watch RMSE Lie

Below is the asymmetric scoring curve, plus a tiny synthetic model controlled by two sliders — bias and noise std. Slide the bias to +10 and the bottom-right scatter goes red. Slide it to -10 and the same magnitude of error suddenly looks safe. Watch RMSE move slightly while NASA score moves a lot.

The asymmetry is not subtle. With $\sigma = 8$ noise, a +6 mean bias roughly doubles the NASA score relative to a 0 mean bias, while RMSE only worsens by 28%. Build a loss function out of RMSE and your gradient does not know that doubling.

Python: RMSE and NASA, Side by Side

Two implementations of NASA score and RMSE; three synthetic models with the same precision but different bias. The NumPy run below produces the exact values quoted in tables throughout this book.

1import numpy as np
2
3# ----- The NASA C-MAPSS scoring function -----
4# Lower is safer. Late predictions (d > 0) cost much more than early ones.
5def nasa_score(predicted: float, actual: float) -> float:
6    d = predicted - actual
7    if d < 0:
8        return np.exp(-d / 13.0) - 1.0   # early - gentle penalty
9    return np.exp( d / 10.0) - 1.0       # late  - exponential penalty
10
11
12def rmse(pred: np.ndarray, actual: np.ndarray) -> float:
13    return float(np.sqrt(np.mean((pred - actual) ** 2)))
14
15
16def total_nasa(pred: np.ndarray, actual: np.ndarray) -> float:
17    return float(sum(nasa_score(p, a) for p, a in zip(pred, actual)))
18
19
20# ----- Three "models" with the same precision but different bias -----
21np.random.seed(0)
22n = 100
23actual = np.random.uniform(20, 120, n)
24
25np.random.seed(0)
26pred_unbiased = actual + np.random.normal(0,  8, n)   # symmetric noise
27
28np.random.seed(0)
29pred_late     = actual + np.random.normal(+6, 8, n)   # biased late
30
31np.random.seed(0)
32pred_early    = actual + np.random.normal(-6, 8, n)   # biased early
33
34
35for name, p in [("unbiased N(0,8) ", pred_unbiased),
36                ("late-biased N(+6,8)", pred_late),
37                ("early-biased N(-6,8)", pred_early)]:
38    print(f"{name:22s}  RMSE = {rmse(p, actual):6.2f}"
39          f"   NASA = {total_nasa(p, actual):8.1f}")
40
41# unbiased N(0,8)         RMSE =   8.08   NASA =     99.3
42# late-biased N(+6,8)     RMSE =  10.34   NASA =    180.1
43# early-biased N(-6,8)    RMSE =   9.77   NASA =    111.4

What the numbers say

The unbiased model A has the lowest RMSE (8.08) and the lowest NASA (99.3). Add a +6-cycle bias and you get model B, with worse RMSE (10.34) and much-worse NASA (180.1). Now flip the sign: the early-biased model C ends up with similar RMSE to B (9.77) but its NASA score is 38% lower (111.4) — because all of C's errors fell on the “safe” side of the curve.

That gap between B and C is the entire research thesis of this book in one number. Two models with comparable RMSE; one is dangerous, the other is safe. The difference is invisible to the metric most papers report.

PyTorch: A Differentiable NASA-Style Loss

For training we need the asymmetric cost as a differentiable $\mathcal{L}$. The shape stays the same; the implementation must use torch.exp and torch.where so autograd can backpropagate through both branches.

1import torch
2import torch.nn as nn
3
4# ----- A differentiable NASA-style loss for training -----
5class AsymmetricRULLoss(nn.Module):
6    """Smoothed NASA-style loss. Lower is better.
7
8    Args:
9        early_scale: divisor in the d < 0 branch (NASA default 13).
10        late_scale:  divisor in the d > 0 branch (NASA default 10).
11    """
12
13    def __init__(self, early_scale: float = 13.0, late_scale: float = 10.0):
14        super().__init__()
15        self.early_scale = early_scale
16        self.late_scale  = late_scale
17
18    def forward(self, pred: torch.Tensor, actual: torch.Tensor) -> torch.Tensor:
19        d = pred - actual                                   # (B,)
20        early = torch.exp(-d / self.early_scale) - 1.0
21        late  = torch.exp( d / self.late_scale ) - 1.0
22        # torch.where keeps autograd flowing through both branches.
23        per_sample = torch.where(d < 0, early, late)
24        return per_sample.mean()
25
26
27# ----- Use it -----
28torch.manual_seed(0)
29B = 8
30actual = torch.rand(B) * 100 + 20      # uniform RUL in [20, 120)
31pred   = actual + torch.randn(B) * 8   # noisy predictions
32
33loss_fn = AsymmetricRULLoss()
34loss    = loss_fn(pred, actual)
35
36print("d        =", (pred - actual).round(decimals=2).tolist())
37print("loss     =", round(loss.item(), 4))
38print("RMSE     =", round(((pred - actual) ** 2).mean().sqrt().item(), 4))

Asymmetric Cost in Other Domains

The pattern “errors in one direction are categorically worse than the other” is everywhere once you start looking.

The mathematical machinery in this book — an asymmetric loss, a gradient-balancing controller, a Pareto-frontier picture — transfers line-for-line to any of these settings.

The Trap: Optimising the Wrong Metric

It is exactly this trap that motivates the failure-biased weighted MSE in Chapter 14. By up-weighting near-failure samples we change the gradient landscape so the model pays disproportionate attention to the regime where late predictions are most expensive — without abandoning the smoothness of MSE that makes deep nets train.

The deep observation. Once you accept that the cost is asymmetric, all three of the proposed objectives in this book (AMNL, GABA, GRACE) follow as different ways to embed that asymmetry into a multi-task gradient.

Takeaway

• RMSE is symmetric; reality is not. A late RUL prediction costs categorically more than an equally-wrong early one. RMSE cannot tell you which side of zero your errors live on.
• The NASA score makes that asymmetry quantitative. $s(d) = e^{-d/13}-1$ for early errors and $s(d) = e^{d/10}-1$ for late ones. Small denominator, exponential growth: lateness compounds fast.
• Two models with the same RMSE can have very different safety profiles. Our late-biased model B beats the early-biased model C on RMSE but loses badly on NASA score (180 vs 111).
• The PyTorch loss is a four-line module. torch.where turns the piecewise definition into a fully differentiable, batched, GPU-friendly forward pass.
• This is what the rest of the book is about. AMNL, GABA, and GRACE are three different engineering answers to the same question: how do we get a model whose gradient knows that late predictions are dangerous?