The RMSE-NASA Pareto Picture

Pareto, Cars, And Why You Cannot Have It All

Vilfredo Pareto, the Italian economist, noticed in 1896 that a policy reform that helps everyone equally is rare; what is far more common is a reform that helps some at the expense of others. The configurations where you cannot improve any individual's outcome without making someone else worse off are Pareto-optimal — the boundary of the attainable region.

Engineering knows this picture well. Tune an F1-car suspension for grip and you lose top speed; tune for top speed and you lose grip. The set of suspensions where no other configuration is BOTH grippier AND faster is the Pareto frontier. A team can choose a point on the frontier based on the day's circuit but cannot beat the frontier without changing something outside the suspension — a different tyre compound, a different aerodynamic package, a different driver.

Section 23·1 reported one number: GRACE wins NASA at 232.7 on multi-condition C-MAPSS. This section shows the picture underneath that number. RMSE and NASA define a 2D performance space; the 9 published MTL methods scatter inside it; only 3 are on the Pareto frontier; GRACE is one of them. The plot is an honest record of which methods to choose, when, and what you get up.

Pareto Dominance In Two Lines

Two-axis Pareto dominance is a precise relation. For two candidate methods $p$ and $q$ with metrics $(r_p, n_p)$ and $(r_q, n_q)$ where lower is better on both axes:

$q \succ p \quad \Longleftrightarrow \quad r_q \leq r_p \;\text{and}\; n_q \leq n_p \;\text{and}\; (r_q < r_p \;\text{or}\; n_q < n_p).$

Read this as ‘$q$ dominates $p$’: q is no worse on either axis AND strictly better on at least one. The strict-better clause excludes ties — if $(r_p, n_p) = (r_q, n_q)$ exactly, neither dominates the other and both are candidates for the front. The Pareto frontier is the set of methods that no other method dominates.

Interactive: The Multi-Condition Pareto Picture

Below: 9 methods plotted in (RMSE, NASA) space, multi-condition mean. The dashed green line traces the Pareto frontier. Drag the preference slider to compose RMSE and NASA into a single score $J = \lambda \cdot \text{RMSE}_{\text{norm}} + (1-\lambda) \cdot \text{NASA}_{\text{norm}}$ and watch the optimal method change. Toggle FD002 / FD004 / multi-cond mean above the chart.

Two things to notice. Drag $\lambda$ from 0 to 1: the optimal method walks along the frontier from GRACE (safety-only) through GABA (balanced) to AMNL (accuracy-only). Switch from MULTI to FD002: the frontier gains Baseline as a fourth corner. Switch to FD004: the frontier collapses to GradNorm alone — one method dominates everything.

Three Methods Make The Multi-Condition Front

The three corners answer different questions. AMNL says: you only care about accuracy (e.g. an HM benchmark scoreboard). GABA says: you accept some accuracy cost to keep the safety score reasonable. GRACE says: safety dominates — buy every possible NASA point, and only stop when accuracy starts to regress.

The 6 dominated methods (Baseline, DWA, GradNorm, Uncertainty, PCGrad, CAGrad) all have at least one front method that beats them on both axes. Uncertainty (RMSE 7.98, NASA 233.8) is the closest to the front: GRACE beats it by 0.06 RMSE and 1.1 NASA — within seed noise. The published claim is therefore ‘GRACE strictly dominates Uncertainty’ with a small statistical margin; section 22·3 §5 explained why this is robust at n=5 seeds.

FD002 Has A Four-Method Front

FD002 is the ‘easy’ multi-condition subset (single fault mode) and the front widens to four methods. Baseline appears here because plain MTL with fixed equal weights already does a reasonable job balancing RMSE and NASA when the data only has one fault to learn. GRACE still wins NASA, GABA is one tick higher in NASA but lower in RMSE, AMNL stretches all the way to the accuracy corner.

FD004 Has A Single Winner

FD004 (multi-condition multi-fault — the hardest C-MAPSS subset) produces a singular winner: GradNorm beats every other method on BOTH axes. GRACE comes second on RMSE (8.12) and second on NASA (242.0) but is dominated by GradNorm. The reason GRACE still wins the multi-condition mean is its FD002 dominance: the 232.7 multi-cond NASA = mean(223.4, 242.0) is lower than any other method's mean of the same pair.

Picking A Point With A Preference Weight

For deployment you need to pick ONE method, not the whole front. The textbook way: define a preference weight $\lambda \in [0, 1]$ capturing how much you care about RMSE relative to NASA, then minimise the composite score

$J(\text{method}) = \lambda \cdot \widetilde{\text{RMSE}} + (1 - \lambda) \cdot \widetilde{\text{NASA}}$

where the tildes denote min-max normalisation onto $[0, 1]$ (so the two axes can be added without unit-scale issues). At $\lambda = 0$ only NASA matters → GRACE wins. At $\lambda = 1$ only RMSE matters → AMNL wins. In between, the optimum walks along the front.

The cut-points are dataset-specific (the explorer above recomputes them live), but the qualitative picture — GRACE for safety-heavy weights, AMNL for accuracy-heavy weights, GABA in between — is robust across all four C-MAPSS subsets and the N-CMAPSS DS02 set covered in chapter 23·3.

Python: Pareto Sorting From Scratch

Implement Pareto-front identification on the 9-method results matrix. The algorithm is two nested loops — O(N²) which is plenty for N=9. Larger problems use NSGA-II's fast-non-dominated-sort which is O(N log N), but the definitions are the same.

1"""Pareto front identification on the GRACE results matrix.
2
3A point p is ‘dominated’ if there is another point q that is
4no worse than p on every axis AND strictly better on at least one.
5The Pareto front is the set of points that are NOT dominated by any
6other.
7"""
8
9import numpy as np
10
11
12# ---------- 9-method results on multi-condition C-MAPSS ----------
13# Source: paper data_analysis/cmapss_h256_complete_140.csv averaged
14# across FD002 + FD004, n=5 seeds per dataset.
15methods = ["Baseline","AMNL","DWA","GABA","GRACE",
16           "GradNorm","Uncertainty","PCGrad","CAGrad"]
17rmse    = np.array([8.06, 7.45, 8.13, 7.89, 7.92,
18                    7.96, 7.98, 8.70, 8.78])
19nasa    = np.array([252.5, 446.7, 251.0, 235.7, 232.7,
20                    241.9, 233.8, 280.6, 282.1])
21
22
23def is_dominated(i, rmse, nasa):
24    """Return True if point i is dominated by any other point."""
25    for j in range(len(rmse)):
26        if i == j:
27            continue
28        if (rmse[j] <= rmse[i] and nasa[j] <= nasa[i]
29                and (rmse[j] < rmse[i] or nasa[j] < nasa[i])):
30            return True
31    return False
32
33
34def pareto_front(rmse, nasa, names):
35    """Return the indices of non-dominated points, sorted by RMSE."""
36    front_idx = [i for i in range(len(rmse))
37                 if not is_dominated(i, rmse, nasa)]
38    # Sort by RMSE ascending — gives the natural left-to-right curve
39    front_idx.sort(key=lambda i: rmse[i])
40    return [(names[i], float(rmse[i]), float(nasa[i])) for i in front_idx]
41
42
43# ---------- Compute and print ----------
44front = pareto_front(rmse, nasa, methods)
45
46print("Pareto front (multi-condition mean):")
47for name, r, n in front:
48    print(f"  {name:<14} RMSE = {r:5.2f}  NASA = {n:6.1f}")
49
50# Confirm GRACE dominates Uncertainty + 5 others
51grace_idx = methods.index("GRACE")
52dominated_by_grace = [
53    methods[j] for j in range(len(methods))
54    if j != grace_idx and rmse[grace_idx] <= rmse[j] and nasa[grace_idx] <= nasa[j]
55       and (rmse[grace_idx] < rmse[j] or nasa[grace_idx] < nasa[j])
56]
57print(f"\nGRACE strictly dominates: {dominated_by_grace}")

PyTorch: Hooking RMSE+NASA Into An Eval Loop

Production usage: take a trained model, run the paper's Evaluator, place the result on the published Pareto picture, and decide whether the new model is publishable. The wrapper returns three signals: the raw $(\text{RMSE},\ \text{NASA})$ pair, a boolean ‘on Pareto front vs paper’, and the lists of paper methods that dominate or are dominated.

{
  'rmse': 7.92,
  'nasa': 232.7,
  'on_front_vs_paper': True,
  'dominators': [],
  'dominated_paper_methods': ['Uncertainty', 'Baseline']
}

1"""Compute RMSE and NASA on a real test set, then place the result.
2
3Reuses the paper's Evaluator from grace/training/evaluator.py.
4The evaluator returns BOTH metrics in a single dict; what we add
5here is the Pareto-positioning logic that turns a single (rmse, nasa)
6result into a recommendation.
7"""
8
9import torch
10from torch.utils.data import DataLoader
11from grace.training.evaluator import Evaluator
12from grace.training.seed_utils import set_seed
13
14
15# Reference points from paper Table I (multi-condition mean)
16REFERENCE = {
17    "AMNL":        (7.45, 446.7),
18    "GABA":        (7.89, 235.7),
19    "GRACE":       (7.92, 232.7),
20    "Uncertainty": (7.98, 233.8),
21    "Baseline":    (8.06, 252.5),
22}
23
24
25def evaluate_and_place(model, test_loader: DataLoader,
26                       device: torch.device, seed: int = 42):
27    """Run the model and report its RMSE+NASA position vs the reference."""
28    set_seed(seed)
29    evaluator = Evaluator(device=device, max_rul=125)
30
31    metrics = evaluator.evaluate(model, test_loader)
32    rmse  = metrics["rmse_last"]
33    nasa  = metrics["nasa_score"]
34
35    # Compare to references via Pareto dominance
36    dominators = []
37    dominated  = []
38    for name, (r_ref, n_ref) in REFERENCE.items():
39        if r_ref <= rmse and n_ref <= nasa and (r_ref < rmse or n_ref < nasa):
40            dominators.append(name)
41        if rmse <= r_ref and nasa <= n_ref and (rmse < r_ref or nasa < n_ref):
42            dominated.append(name)
43
44    on_front = len(dominators) == 0
45    return {
46        "rmse": rmse, "nasa": nasa,
47        "on_front_vs_paper": on_front,
48        "dominators": dominators,
49        "dominated_paper_methods": dominated,
50    }
51
52
53# ---------- Stub usage ----------
54if __name__ == "__main__":
55    pass
56    # In production:
57    #   model = load_checkpoint("...")
58    #   loader = make_test_loader("FD002")
59    #   result = evaluate_and_place(model, loader, torch.device("cuda"))
60    #   print(result)

Pareto Tradeoffs In Other Domains

In every row the ‘publish your Pareto front, not your winner’ norm is becoming standard. Single-number comparisons (mAP, MMLU, Dice) hide deployment trade-offs the practitioner needs to see. GRACE's contribution to the C-MAPSS conversation is the same: shift the literature from ‘best RMSE wins’ to ‘here is the Pareto picture, here is where we sit, here is what you give up if you choose differently’.

Pitfalls When Reading A Pareto Plot

Pitfall 1: confusing ‘on the front’ with ‘the winner’

AMNL is on the front because nothing dominates it on RMSE; that does NOT make AMNL a good general-purpose method. Its NASA score is twice GRACE's. Being on the front means ‘optimal for SOME preference’, not ‘best overall’.

Pitfall 2: averaging across datasets before Pareto-ising

The multi-condition mean has 3 methods on the front; FD004 alone has 1 (GradNorm), FD002 alone has 4. Per-dataset and averaged fronts can disagree dramatically. Always report per-dataset Pareto pictures alongside the averaged one — the averaging hides reversals.

Pitfall 3: ignoring seed variance

On FD002 GRACE's NASA = 223.4 ± 26.5 (one-sigma) and Uncertainty's NASA = 224.4 ± 35.3. The two error bars overlap; the ‘GRACE strictly dominates Uncertainty’ claim is a POINT-ESTIMATE claim, not a statistical one. Always report SEM bars on Pareto plots; the paper's Fig. 3 does.

Pitfall 4: choosing λ post-hoc

Picking the preference weight $\lambda$ AFTER seeing the results — e.g. ‘$\lambda = 0.3$ happens to make my method win’ — is post-hoc cherry-picking. The$\lambda$ must come from the deployment context (operational cost ratios, regulatory requirements) not from a sweep that targets a desired winner.

Pitfall 5: forgetting that lower is better on both axes here

Some Pareto plots in the literature put accuracy on the x-axis (higher better) and cost on the y-axis (lower better). The frontier then sits in the upper-right. The C-MAPSS Pareto plot has both axes in ‘lower better’ orientation; the frontier sits in the lower-left. Read the axis legends before interpreting; mirroring an axis silently inverts the front.

Takeaway

• Pareto dominance is precise: $q \succ p$ iff q is no worse on either axis AND strictly better on at least one. The Pareto front is the set of methods that no other method dominates.
• On multi-condition C-MAPSS the front has 3 methods: AMNL (accuracy corner), GABA (middle), GRACE (safety corner). On FD002 alone the front widens to 4 (Baseline added). On FD004 alone it collapses to 1 (GradNorm dominates everything).
• GRACE strictly dominates 6 of the 8 other methods on multi-condition mean: Baseline, DWA, GradNorm, Uncertainty, PCGrad, CAGrad. It does NOT dominate AMNL or GABA — those have lower RMSE.
• Picking a deployment point uses a preference weight $\lambda \in [0, 1]$: GRACE for $\lambda \leq 0.4$ (safety-first), GABA for the middle, AMNL for $\lambda \geq 0.85$ (accuracy-first).
• The Pareto-front norm is now standard across deep learning: object detection, LLM serving, medical imaging, energy, recommenders. Always publish the front, not the winner.