Hyperparameters and Defaults

Wine, Grapes, And Hyperparameters

A bottle of Burgundy and a bottle of Beaujolais can be made from the same Pinot Noir grapes harvested from neighbouring hillsides. What separates them is the recipe: yeast strain, fermentation temperature, time on lees, oak ageing. The grapes set the ceiling; the recipe sets the floor. Same logic in machine learning: the architecture sets the ceiling on what the model can represent, and the hyperparameters set the floor on what it actually does.

GRACE's architecture has been fixed since chapter 8 (CNN + BiLSTM + multi-head attention + FC). The published 7.72 RMSE on FD002 depends on every hyperparameter being at its production default. This section is the catalogue: every default, every value's sensitivity (from the paper's ablation), and the search protocol the authors used to pick them. Section 22.3 gives the reproducibility recipe, but it cannot reproduce anything if you don't know which knob to leave alone.

Two Distinct Categories: Architecture vs. Training

Hyperparameters in any deep-learning system fall into two categories with very different sensitivity profiles:

The split matters because the search budgets are different. Architecture sweeps need full-data training (~30 min per trial × 5 seeds = 2.5 hours per point), so the paper sweeps only a handful of named configs (cmapss, ncmapss_20feat, ncmapss_32feat). Training sweeps run 4–5 candidates per axis along coordinate-descent, totalling ~20 hours for all axes on FD002.

Architecture Defaults

The C-MAPSS production architecture, from grace/models/model_configs.py:"cmapss":

Two siblings exist for N-CMAPSS: ncmapss_20feat (input_size=20, hidden_size=256, 8 heads — matches the DKAMFormer protocol) and ncmapss_32feat (input_size=32, hidden_size=512, 16 heads — uses all sensors plus operating conditions for higher capacity).

Training Defaults

Production training hyperparameters, from grace/experiments/config.py:ExperimentConfig:

GABA-Specific Defaults And Robustness

The OUTER controller adds three of its own knobs:

The remarkable property of GABA documented in the paper's Fig. 6: the converged $\lambda_{\text{health}}$ sits in $[0.985,\ 1.001]$ across all four C-MAPSS subsets and all five seeds (n=20 runs total). Standard deviation is $\sigma \approx 0.003$. This is the OUTER axis being self-calibrating: regardless of which dataset you put in, GABA finds the same equilibrium because the gradient ratio is structurally similar (450–650× across all subsets — chapter 21 §3 Pareto explorer).

Interactive: Hyperparameter Impact On RMSE

Click any row to see the description and the GRACE recommendation. The bars show RMSE on FD002 (left column) and FD004 (right column) relative to the V7 baseline. Green = better than baseline, red = worse. Numbers come from the paper's data_analysis/comprehensive_analysis.py ablation block.

What stands out. Removing the health task (Single-Task) is catastrophic (+260% RMSE) — the auxiliary is doing real regularisation work. Removing attention costs ~13%. Switching from WMSE to plain MSE swaps the dataset winners (FD002 worsens, FD004 improves) — a useful diagnostic when GRACE underperforms on a new dataset. Most other choices are within 1–5% of baseline; the architecture and the choice of multi-task learning are what matter most.

How The Defaults Were Found: A Search Protocol

The paper authors did not run a full 18-dimensional grid search — that would be 3¹⁸ ≈ 387 million trials at 30 minutes each. Instead the protocol was sequential, prioritised by sensitivity:

• Step 1: literature defaults. AdamW lr=1e-3, β=(0.9, 0.999), weight_decay=1e-4, batch_size=256. These match transformer-era literature. No tuning yet.
• Step 2: architecture sweep. Four named configs (full / lstm_only / transformer / mlp / cnn_only) with GABA on FD001+FD002, 5 seeds each. Locks in CNN + BiLSTM + Attention as the backbone (chapter 27).
• Step 3: GABA controller sweep. β ∈ {0.9, 0.99, 0.999}, ε_floor ∈ {0.01, 0.05, 0.10}, warmup_steps ∈ {0, 100, 500}. 27-trial grid on FD002 only (single-dataset is enough — chapter 22 §2 robustness). Locks in (0.99, 0.05, 100).
• Step 4: training-cadence coordinate descent. One axis at a time: warmup_epochs, scheduler_patience, ema_decay, grad_clip. Single-axis sweeps because the surfaces are approximately separable. Total ~20 trials.
• Step 5: weight-decay schedule (GRACE-only). Three options compared: fixed 1e-4, halve-at-100, schedule (1e-4 → 5e-5 → 1e-5). The schedule won by 0.05 RMSE on FD002; adopted as default.
• Step 6: 5-seed cross-validation. Final configuration run with 5 seeds on all four C-MAPSS subsets + N-CMAPSS DS02. Reported numbers are 5-seed means.

Total compute: ~150 trials × 30 min = 75 hours of GPU time. A full grid search would have needed millions; coordinate descent and judgement reduce the budget by four orders of magnitude.

Python: Coordinate-Descent Hyperparameter Search

A self-contained NumPy reproduction of the search protocol on a toy RMSE landscape that mirrors the paper's actual landscape on FD002. Grid search uses 27 trials; coordinate descent finds the same optimum with 9. Read the code line by line to see how the dict-spread + min-with-key idioms encode coordinate descent in three lines.

Grid search:        27 trials → RMSE 8.0000 at (0.001, 256, 0.999)
Coordinate descent: 9 trials → RMSE 8.0000 at (0.001, 256, 0.999)
Speedup:            3.0× — same optimum, less compute

1"""Hyperparameter search by coordinate descent — what the paper did.
2
3Compares grid search (27 trials) with coordinate descent (9 trials).
4Both find the same optimum on a toy RMSE surface that mimics the
5landscape the GRACE authors saw on FD002.
6"""
7
8import numpy as np
9
10
11def toy_rmse(lr, bs, beta):
12    """Synthetic RMSE landscape with a sharp minimum at (1e-3, 256, 0.999).
13
14    Mirrors the paper's observation: lr is fairly forgiving (broad
15    minimum), batch size has a mild slope, and EMA beta has a sharp
16    minimum at 0.999 — outside that band the EMA either tracks too
17    slowly (0.9999) or too fast (0.99) for the late-training noise.
18    """
19    lr_term   = 5.0 * ((np.log10(lr) + 3) / 2) ** 2
20    bs_term   = 0.5 * ((np.log2(bs) - np.log2(256)) / 1.0) ** 2
21    beta_term = 2.0 * ((beta - 0.999) / 0.001) ** 2
22    return 8.0 + lr_term + bs_term + beta_term
23
24
25# ---------- Search space ----------
26LRS   = [1e-4, 1e-3, 1e-2]
27BSS   = [64, 128, 256]
28BETAS = [0.99, 0.999, 0.9999]
29
30
31# ---------- Grid search (27 trials) ----------
32def grid_search():
33    best, best_r = None, np.inf
34    n_trials = 0
35    for lr in LRS:
36        for bs in BSS:
37            for beta in BETAS:
38                r = toy_rmse(lr, bs, beta)
39                n_trials += 1
40                if r < best_r:
41                    best, best_r = (lr, bs, beta), r
42    return best, best_r, n_trials
43
44
45# ---------- Coordinate descent (9 trials) ----------
46def coordinate_descent(lr0=1e-3, bs0=128, beta0=0.99):
47    cur = {"lr": lr0, "bs": bs0, "beta": beta0}
48    n_trials = 0
49    for axis, candidates in [("lr", LRS), ("bs", BSS), ("beta", BETAS)]:
50        # Sweep ONLY this axis; freeze the others
51        scores = [(c, toy_rmse(**{**cur, axis: c})) for c in candidates]
52        n_trials += len(scores)
53        cur[axis] = min(scores, key=lambda t: t[1])[0]
54    best = (cur["lr"], cur["bs"], cur["beta"])
55    return best, toy_rmse(*best), n_trials
56
57
58# ---------- Run both ----------
59g_best, g_rmse, g_n = grid_search()
60c_best, c_rmse, c_n = coordinate_descent()
61
62print(f"Grid search:        {g_n} trials → RMSE {g_rmse:.4f} at {g_best}")
63print(f"Coordinate descent: {c_n} trials → RMSE {c_rmse:.4f} at {c_best}")
64print(f"Speedup:            {g_n / c_n:.1f}× — same optimum, less compute")

PyTorch: Hyperparameter Scoping In ExperimentConfig

Production search uses the paper's ExperimentConfig dataclass + a generic sweep_axis() helper. Every result is a (value, mean_rmse) tuple recoverable from disk. Reproducibility is built in — the dataclass serialises to JSON and back, so a sweep can be paused, audited, and resumed.

LR    sweep: [(0.0003, 8.2), (0.0005, 7.9), (0.001, 7.72), (0.002, 8.1), (0.005, 9.6)]
EMA   sweep: [(0.99, 8.0), (0.995, 7.85), (0.999, 7.72), (0.9995, 7.78)]
BS    sweep: [(64, 7.95), (128, 7.78), (256, 7.72), (512, 7.85)]

1"""Production hyperparameter scope: ExperimentConfig + sweep loop.
2
3The paper centralises every default in a single dataclass at
4grace/experiments/config.py. Sweeping a hyperparameter means
5overriding ONE field and rerunning. This pattern keeps the search
6reproducible: every result is a (config_dict, rmse) tuple that can
7be re-run from disk.
8"""
9
10from dataclasses import dataclass, field, replace
11from typing import List
12from grace.experiments.phase1_cmapss import run_single_experiment
13
14
15@dataclass
16class ExperimentConfig:
17    """Mirror of grace/experiments/config.py with the production defaults."""
18    dataset:       str   = "FD002"
19    mtl_method:    str   = "gaba"
20
21    # Training schedule
22    epochs:                int   = 500
23    batch_size:            int   = 256
24    lr:                    float = 1e-3
25    patience:              int   = 80
26    grad_clip:             float = 1.0
27    use_ema:               bool  = True
28    ema_decay:             float = 0.999
29    warmup_epochs:         int   = 10
30    scheduler_factor:      float = 0.5
31    scheduler_patience:    int   = 30
32    min_lr:                float = 5e-6
33    weight_decay:          float = 1e-4
34
35    # Data
36    sequence_length:       int   = 30
37    per_condition_norm:    bool  = False
38
39    # GRACE-specific
40    use_weighted_mse:        bool  = True
41    adaptive_weight_decay:   bool  = True
42    initial_weight_decay:    float = 1e-4
43
44    # Reproducibility
45    seeds: List[int] = field(default_factory=lambda: [42, 123, 456, 789, 1024])
46
47
48# ---------- Coordinate-descent sweep helper ----------
49def sweep_axis(base_cfg: ExperimentConfig, axis: str, candidates: list):
50    """Run the same training pipeline once per candidate value on one axis.
51
52    Args:
53        base_cfg:    Production-default ExperimentConfig.
54        axis:        Field name to override (e.g. 'lr', 'ema_decay').
55        candidates:  Values to try for that field.
56
57    Returns:
58        List of (value, mean_rmse) tuples — averaged over base_cfg.seeds.
59    """
60    results = []
61    for value in candidates:
62        cfg = replace(base_cfg, **{axis: value})
63        seed_rmses = []
64        for seed in cfg.seeds:
65            r = run_single_experiment(cfg, seed)
66            seed_rmses.append(r["best_rmse"])
67        mean_rmse = sum(seed_rmses) / len(seed_rmses)
68        results.append((value, mean_rmse))
69    return results
70
71
72# ---------- Run the three production sweeps ----------
73base = ExperimentConfig(dataset="FD002")
74
75lr_results        = sweep_axis(base, "lr",         [3e-4, 5e-4, 1e-3, 2e-3, 5e-3])
76ema_results       = sweep_axis(base, "ema_decay",  [0.99,  0.995, 0.999, 0.9995])
77bs_results        = sweep_axis(base, "batch_size", [64, 128, 256, 512])
78
79print("LR    sweep:", lr_results)
80print("EMA   sweep:", ema_results)
81print("BS    sweep:", bs_results)

Adapting These Defaults To Other Domains

The pattern is consistent: training cadence values (warmup, scheduler, EMA) generalise within ±2× without retuning. Architecture-specific values (sequence_length, hidden_size, n_heads) need a per-domain sweep because they interact with the data's temporal/spatial structure. Lr and batch_size sit in between — transferable as orders of magnitude but not as exact values.

Pitfalls When Tuning Hyperparameters

Pitfall 1: tuning on the test set

The C-MAPSS ‘test’ set has only 100–260 units. Sweeping multiple training hyperparameters and reporting the best test rmse_last leaks information. The paper avoids this by holding out a 10% validation slice from training (chapter 7 §3) for sweep decisions; test numbers are computed only at the very end on the locked configuration.

Pitfall 2: insufficient seeds for a sweep decision

Single-seed differences of 0.3 RMSE on FD002 are within the seed-to-seed noise. Always average at least 3 seeds before choosing between two candidates; the paper used 5. A two-seed difference of 1.0 RMSE may still be noise — check the per-seed std (chapter 22 §3 reproducibility).

Pitfall 3: optimising for RMSE only

AMNL (cell B from chapter 21 §1) wins RMSE 6.74 on FD002 but loses NASA 356. If the production goal is safety-aware prediction, optimising only on RMSE picks the wrong winner. Always run the search on the metric that matches the deployment objective — or jointly on a Pareto frontier.

Pitfall 4: assuming separability across categories

Architecture and training are not separable when moving between datasets with very different scale (C-MAPSS ~4M windows vs. tabular toy datasets ~1k rows). Doubling hidden_size on a small dataset hurts because the lr and batch_size have to come down too. Always sweep architecture first; only after locking it should you tune training.

Pitfall 5: not recording the random seeds

Reproducing a result needs the seeds used for both the data split and the model init. The paper records both via set_seed(seed) and writes the seed into the output directory name. A ‘best result’ without its seed is not reproducible.

Takeaway

• GRACE has 18 hyperparameters split between architecture (10, in model_configs.py) and training (8+, in experiments/config.py) plus 3 GABA-specific knobs.
• The published values are 5-seed validated; the GABA controller is empirically robust across all four C-MAPSS subsets (converged $\lambda_{\text{health}} = 0.99 \pm 0.008$).
• The largest sensitivity directions are the loss formulation (single-task is +260% RMSE; no attention is +13%); most training-cadence hyperparameters move RMSE by <5%.
• The search protocol is coordinate descent, not grid: ~150 trials, ~75 GPU-hours total. Grid search would need 387M.
• Adapt to a new domain by sweeping architecture first, then training. Training-cadence values (warmup, scheduler, EMA) generalise within ±2× without retuning.