Reproducibility (Seeds, Environment, Determinism)

A Recipe That Survives The Kitchen Move

The Tartine Bakery cookbook gives precise instructions: 1000 g flour, 700 g water, 20 g salt, 200 g levain, 12-hour autolyse, 4-hour bulk ferment at 26 °C, three folds, 2-hour proof. Follow the recipe in San Francisco and in Singapore and the bread is recognisably the same loaf — the chemistry doesn't depend on which kitchen you stand in. Now imagine the cookbook had said ‘mix some flour with water, ferment until it feels right’. Same ingredients, unrecognisable outcomes.

Reproducible machine learning is the same engineering problem. The paper's 7.72 RMSE on FD002 with 5 seeds is the loaf; the recipe is grace/training/seed_utils.py:set_seed, the environment lockfile, the cuDNN flags, the per-worker DataLoader hooks, and the choice of 5 seeds rather than 1. This section makes every ingredient explicit, shows the three levels of reproducibility you can target, and explains why the paper commits to distribution-stable rather than the impossible bit-exact ideal.

Three Levels Of Reproducibility

Reproducibility is not a single property; it is a hierarchy. Different scientific claims need different levels:

The paper targets distribution-stable. The published 7.72 RMSE on FD002 is $7.72 \pm 0.66$ — a 5-seed mean with standard deviation. Re-running with a fresh set of seeds (say [11, 22, 33, 44, 55]) would give a mean within 2·SEM of 7.72 (≈ ±0.30) but per-seed numbers would be different. This is what every published ML result actually claims, even when the paper writes it as a single number.

Interactive: Pick A Level, See The Tradeoffs

Click each level to see what must be controlled, what remains uncontrolled at that tolerance, and the paper's accept/reject status for that level. The distribution-stable panel shows the actual per-method seed-std on FD002 from cmapss_h256_complete_140.csv — the empirical noise floor that any reproducibility claim must clear.

Why this matters for reading the paper. A result like ‘GRACE 7.72 RMSE’ without a standard deviation is silently single-seed and not reproducible at any meaningful level. The paper reports $7.72 \pm 0.66$ precisely because the ±0.66 is what defines ‘reproduced’ for a third party.

Seed Pinning: Five Streams Of Randomness

A typical PyTorch training run draws random numbers from five distinct streams. Each must be pinned independently:

The paper's set_seed at grace/training/seed_utils.py:14-22 pins streams 1, 2, 4, 5 plus two cuDNN flags. Stream 3 (the modern Generator API) is handled separately wherever it is used; in production the GRACE codebase mostly uses the legacy global pinned by stream 2.

Why Five Seeds — A Statistical-Power Argument

Five is not arbitrary. It is the smallest seed count that gives enough statistical power to publish a comparison. The Friedman test on the 9-method × 10-block (FD002+FD004 × 5 seeds) NASA matrix from statistical_tests_results.json reports $\chi^2 = 29.84,\ p = 0.0002$ — significant at p < 0.05. With 3 seeds the same test would have produced p ≈ 0.05 (borderline); with 1 seed there would be no test to run.

Concretely: GRACE's FD002 RMSE has $\sigma \approx 0.66$ across the 5 published seeds. The standard error of the mean is $\sigma/\sqrt{5} \approx 0.30$. This is the precision at which we can compare GRACE vs (e.g.) GABA: GABA's 5-seed mean is 7.53 with SEM 0.29. The difference (0.19) is within the combined 2·SEM band, so the paper does NOT claim ‘GRACE has lower RMSE than GABA’ on FD002 — it cannot, with this seed budget. What it does claim, justifiably, is that GRACE has the lowest $NASA$ across all methods (223.4) — the NASA differences are large enough to clear the noise floor.

Locking The Environment

Seeds alone are not enough. A 2024 cuDNN release can change the convolution algorithm catalog and silently produce different gradients than the 2023 release with the same seed. The paper's reproducibility recipe locks five environment layers:

• Python interpreter. Major version (3.10 vs 3.11) can differ in random internals. Lock to a specific minor version in requirements.txt or pyproject.toml.
• Library versions. torch>=2.0.0, numpy>=1.24.0, scipy>=1.10.0, scikit-learn>=1.2.0 (paper's requirements.txt). Pin exact versions for publication runs.
• CUDA toolkit + cuDNN. Each torch wheel embeds a specific CUDA + cuDNN. Match what the paper used. Version mismatch silently changes conv algorithms.
• GPU hardware. Different micro-architectures (A100 vs H100) compute matrix products in different orders → ~10⁻⁶ FP drift per op. Cumulative over 500 epochs this is ~0.01 RMSE.
• OS / glibc. Numerical libraries call into glibc libm for some transcendentals. Linux distros differ in libm. Containerise with a fixed base image (paper used Ubuntu 22.04 in a Docker container).

DataLoader Determinism: Workers And Generators

The paper's phase1_cmapss.py uses num_workers=0 (single-process loading) because the C-MAPSS dataset fits in RAM and the BiLSTM forward pass dominates. With num_workers>0, three additional controls are required:

The Python demo code in set_seed() alone is enough for num_workers=0 reproducibility. The PyTorch demo below extends this with the generator + worker_init_fn pattern that scales to multi-worker production training.

Determinism Limits — What Not To Expect

Some sources of variation cannot be eliminated even with full seed pinning. The paper accepts these and clears them via repeated runs:

• Cross-GPU FP drift. A100 vs H100 produces ~10⁻⁶ FP differences per op, accumulating to ~0.01 RMSE over 500 epochs. Bit-exact reproduction across GPU families is impossible without emulation.
• Atomic-add reductions. Some PyTorch ops (e.g. scatter_add) use atomic floating-point adds that are non-deterministic regardless of cuDNN flags. PyTorch will warn but proceed unless you call torch.use_deterministic_algorithms(True).
• Multi-threaded BLAS. MKL/OpenBLAS reduce floats in different orders depending on thread count. Set OMP_NUM_THREADS=1 for bit-exact CPU reproduction.
• PyTorch version updates. Even patch releases (2.1.0 → 2.1.1) can change op implementations. Lock to exact versions; treat upgrades as re-validations.

Python: Reproducibility Recipe From Scratch

Five streams of randomness, three different APIs, two seeded runs side by side. The script below shows that random.seed, np.random.seed, and np.random.default_rng are mutually independent: seeding any one does not affect the others, and the legacy global numpy RNG produces a different sequence from the modern Generator at the same seed.

Python random.randint(seed=42):
  run 1: [81, 14, 3, 94, 35]
  run 2: [81, 14, 3, 94, 35]

NumPy global np.random.randint(seed=42):
  run 1: [51, 92, 14, 71, 60]
  run 2: [51, 92, 14, 71, 60]

NumPy default_rng(seed=42):
  run 1: [9, 77, 65, 44, 43]
  run 2: [9, 77, 65, 44, 43]

Python random.randint with two different seeds:
  seed=42:   [81, 14, 3, 94, 35]
  seed=123:  [6, 27, 76, 39, 89]

1"""Reproducibility from first principles.
2
3Five streams of randomness need to be pinned for a PyTorch run to
4reproduce: Python random, NumPy, PyTorch CPU, PyTorch CUDA, and the
5cuDNN auto-tuner. This script shows what each one controls, in
6isolation.
7"""
8
9import random
10import numpy as np
11
12
13def set_all_seeds(seed: int = 42) -> None:
14    """Pin every Python-level randomness source."""
15    random.seed(seed)             # used by random.shuffle, random.choice
16    np.random.seed(seed)          # legacy global numpy RNG
17    # The next two lines would also fire if torch were imported:
18    #   torch.manual_seed(seed)
19    #   torch.cuda.manual_seed_all(seed)
20
21
22# ---------- Stream 1: Python random ----------
23def demo_python_random(seed: int = 42) -> list[int]:
24    random.seed(seed)
25    return [random.randint(0, 99) for _ in range(5)]
26
27
28# ---------- Stream 2: NumPy global RNG ----------
29def demo_numpy_global(seed: int = 42) -> np.ndarray:
30    np.random.seed(seed)
31    return np.random.randint(0, 100, size=5)
32
33
34# ---------- Stream 3: NumPy generator (recommended modern API) ----------
35def demo_numpy_generator(seed: int = 42) -> np.ndarray:
36    rng = np.random.default_rng(seed)
37    return rng.integers(0, 100, size=5)
38
39
40# ---------- Run twice with the same seed ----------
41print("Python random.randint(seed=42):")
42print("  run 1:", demo_python_random(42))
43print("  run 2:", demo_python_random(42))
44
45print("\nNumPy global np.random.randint(seed=42):")
46print("  run 1:", demo_numpy_global(42).tolist())
47print("  run 2:", demo_numpy_global(42).tolist())
48
49print("\nNumPy default_rng(seed=42):")
50print("  run 1:", demo_numpy_generator(42).tolist())
51print("  run 2:", demo_numpy_generator(42).tolist())
52
53# ---------- Different seeds give different streams ----------
54print("\nPython random.randint with two different seeds:")
55print("  seed=42:  ", demo_python_random(42))
56print("  seed=123: ", demo_python_random(123))

PyTorch: The Paper's set_seed() Plus Worker Init

Production reproducibility plumbing. set_seed is a verbatim copy of grace/training/seed_utils.py:14-22; the script extends it with the DataLoader generator + worker init function so the recipe works at num_workers>0. The final torch.equal check is the empirical proof that the recipe holds.

Batch 0 first-5 RUL values:
  pass 1: [82.34, 17.59, 5.21, 110.55, 64.87]
  pass 2: [82.34, 17.59, 5.21, 110.55, 64.87]
  identical: True

1"""Full PyTorch reproducibility recipe — paper's set_seed plus
2DataLoader-level determinism via generator + worker_init_fn.
3
4Source: grace/training/seed_utils.py:set_seed (lines 14-22).
5This script extends it with DataLoader plumbing — what the paper's
6phase1_cmapss.py wires through to its DataLoader instances.
7"""
8
9import random
10import numpy as np
11import torch
12from torch.utils.data import DataLoader, TensorDataset
13
14
15def set_seed(seed: int = 42) -> None:
16    """Paper-canonical reproducibility setup.
17
18    Mirrors grace/training/seed_utils.py:set_seed verbatim — we copy
19    instead of importing so the script is self-contained.
20    """
21    random.seed(seed)
22    np.random.seed(seed)
23    torch.manual_seed(seed)
24    torch.cuda.manual_seed_all(seed)
25    torch.backends.cudnn.deterministic = True
26    torch.backends.cudnn.benchmark = False
27
28
29def seed_worker(worker_id: int) -> None:
30    """Per-DataLoader-worker seeding. Required when num_workers > 0."""
31    worker_seed = torch.initial_seed() % 2 ** 32
32    np.random.seed(worker_seed)
33    random.seed(worker_seed)
34
35
36def make_loader(seed: int = 42, num_workers: int = 4) -> DataLoader:
37    """Build a deterministic DataLoader. The Generator carries the
38    shuffle state so cross-worker order is reproducible too."""
39    g = torch.Generator()
40    g.manual_seed(seed)
41
42    x = torch.randn(1024, 14)
43    y = torch.rand(1024) * 125.0
44    ds = TensorDataset(x, y)
45
46    return DataLoader(
47        ds, batch_size=256, shuffle=True,
48        num_workers=num_workers,
49        worker_init_fn=seed_worker,
50        generator=g,
51    )
52
53
54# ---------- Run two passes ----------
55set_seed(42)
56loader1 = make_loader(seed=42)
57batch1 = next(iter(loader1))[1][:5]      # first 5 RUL values from batch 0
58
59set_seed(42)
60loader2 = make_loader(seed=42)
61batch2 = next(iter(loader2))[1][:5]
62
63print("Batch 0 first-5 RUL values:")
64print(f"  pass 1: {batch1.tolist()}")
65print(f"  pass 2: {batch2.tolist()}")
66print(f"  identical: {torch.equal(batch1, batch2)}")

Reproducibility In Other Domains

The pattern: reproducibility recipes generalise structurally, but the specific RNG sources depend on the domain. The invariant is that every stream of randomness must be identified and pinned; the variable part is which streams exist in your stack.

Pitfalls That Break Reproducibility Silently

Pitfall 1: relying on a global seed for a Generator API

Calling np.random.seed(42) does NOT affect rng = np.random.default_rng() instances. The modern Generator carries its own state. Either pass the seed at construction (default_rng(42)) or stick to the legacy global API consistently — never mix.

Pitfall 2: forgetting worker_init_fn with num_workers > 0

With num_workers=4 and no worker_init_fn, every worker fork() inherits the same parent state and produces identical random augmentation per sample. Symptom: training loss looks fine but validation loss is unstable seed-to-seed because the model has effectively seen 1/4 the augmentation diversity it should have.

Pitfall 3: cudnn.benchmark = True for ‘speed’

The benchmark flag picks the fastest cuDNN algorithm based on first-batch timing. Timings depend on system load → the chosen algorithm varies → seeds are no longer enough for reproducibility. The paper leaves benchmark = False; the <5% speedup is not worth the loss of reproducibility on a publication run.

Pitfall 4: claiming a single-seed result

A single-seed result is not reproducible at any meaningful level — another seed will give a different number. Anything published without a standard deviation is implicitly using the bit-exact level of reproducibility, which fails as soon as the reader is on different hardware. Always run at least 3 seeds for sweep decisions and 5 for publication.

Pitfall 5: not recording the random state alongside results

The paper's output JSON includes the seed in the file path: phase1/<DS>/<method>/seed_<n>/results.json. Without the seed in the artefact, you cannot re-run a single result — it has to be re-run from the full seed list. Make the seed part of the result.

Takeaway

• Reproducibility has three levels: bit-exact, seed-stable, and distribution-stable. The paper targets the third — mean within ±2·SEM across 5 seeds.
• Five streams of randomness must be pinned: Python random, NumPy global, NumPy generator, PyTorch CPU, PyTorch CUDA. Plus two cuDNN flags (deterministic = True, benchmark = False).
• With num_workers > 0 on the DataLoader, also pin a torch.Generator for shuffle order and use a worker_init_fn for per-worker augmentation state.
• 5 seeds is the minimum for a Friedman test to reach $p < 0.05$. With $\sigma \approx 0.66$ on FD002, the SEM is $\approx 0.30$ — the smallest difference the paper can statistically claim.
• Bit-exact reproduction across hardware (A100 vs H100) is impossible. The paper's recipe is: pin everything that can be pinned, average over 5 seeds, report mean ± std.