Minimum Floor and Renormalization

The Pressure-Relief Valve

Industrial boilers have a small spring-loaded valve on top. 99% of the time it does nothing. The boiler operates far below the pressure threshold and the valve is invisible. But on the rare day the controller mis-reads a sensor and pressure climbs toward the rupture limit, the valve flicks open and dumps steam — preventing a catastrophic failure that no controller logic could recover from in time. The valve is cheap, simple, and always armed.

GABA needs the same component. The closed form $\lambda^*_i = g_j / (g_i + g_j)$ is mathematically beautiful but operationally brittle: if $g_{\text{health}}$ ever drops to zero (perfect classification on a batch), $\lambda_{\text{rul}}$ snaps to 0 and the regression head stops receiving any gradient. The optimiser silently abandons one task. The floor is the pressure-relief valve.

What Happens Without A Floor

Run a 1,000-step training trace with the §17.3 closed form and §18.2 EMA but NO floor. Three pathologies emerge, any of which silently degrade training:

• Catastrophic suppression. A streak of batches where $g_{\text{health}}$ is unusually small (e.g. all 64 samples are easy ‘Normal’ class) drives $\hat{\lambda}_{\text{rul}}$ near zero. The next 100+ steps see no RUL gradient. When the easy streak ends, the regression head has regressed.
• Asymmetric recovery. Once $\hat{\lambda}_{\text{rul}}$ is near zero, the EMA can only climb back at rate (1−β) per step. With β=0.99, recovering from $\hat{\lambda}_{\text{rul}} = 0.0001$ to $0.01$ takes ~460 steps. That is a permanent cost paid in the loss landscape.
• Noise amplification. $\lambda^*$ near zero is in a regime where the closed form $g_{\text{health}} / (g_{\text{rul}} + g_{\text{health}})$ has very high relative sensitivity (§17.3 sensitivity analysis: derivative ~0.2 when g_health is small). Near-zero weights amplify per-batch fluctuations and increase the effective variance the EMA has to fight.

The floor short-circuits all three failure modes by guaranteeing that every task always has at least $\lambda_{\min}/(1+\lambda_{\min}) \approx \lambda_{\min}$ weight, regardless of the EMA value. No streak of bad batches can fully suppress a task.

The Floor Formula (Paper Eq. 6)

The two-line transformation:

$\text{(floor)} \quad c_i = \max(\hat{\lambda}_i, \lambda_{\min}) \qquad \text{(renormalise)} \quad \lambda^*_i = c_i \Big/ \sum_{j=1}^{K} c_j$

Floor takes any weight below $\lambda_{\min}$ and lifts it to $\lambda_{\min}$; weights above the floor pass through unchanged. After flooring, the sum can exceed 1 (clamping only INCREASES values), so the renormalisation step divides by the new sum to put the result back on the simplex. Together the two operations preserve the property $\sum_i \lambda^*_i = 1$ while bounding $\lambda^*_i$ away from zero.

Anti-Windup: A Control-Theoretic Lens

In classical PID control, ‘windup’ happens when an integrator accumulates error during a period when the actuator is saturated. By the time the actuator un-saturates, the integrator holds a huge accumulated value that takes seconds to discharge — producing large overshoot and oscillation. The fix is anti-windup: clamp the integrator to a bounded range so the accumulator can never blow up.

GABA's EMA $\hat{\lambda}_i$ is functionally an integrator. Without a floor, it can drift toward zero on a long bad streak. The floor is the anti-windup clamp: the integrator state is forbidden from leaving $[\lambda_{\min}, 1 - (K-1) \lambda_{\min}]$. The paper (main.tex:387, 688) explicitly draws this analogy: ‘The floor $\lambda_{\min} = 0.05$ acts as an anti-windup mechanism ensuring no task is fully suppressed.’

The Bounded-Weight Guarantee

Algebraically, the floor + renormalisation guarantees a closed-form bound on the output. For K=2:

$\lambda^*_i \in \left[ \frac{\lambda_{\min}}{1 + \lambda_{\min}}, \; \frac{1}{1 + \lambda_{\min}} \right]$

For $\lambda_{\min} = 0.05$ the bound evaluates to $[0.04762, 0.95238]$ — within rounding of the paper's loose statement $[\lambda_{\min}, 1 - \lambda_{\min}] = [0.05, 0.95]$. The Python demo below verifies this saturation point exactly. The general K-task bound is:

$\lambda^*_i \in \left[ \frac{\lambda_{\min}}{1 + (K - 1) \lambda_{\min}}, \; \frac{1}{1 + (K - 1) \lambda_{\min}} \right]$

Why $\lambda_{\min} = 0.05$

The paper's value of 0.05 is empirical. Three considerations drove the choice:

The paper's hyperparameter robustness section (paper §5.8) reports that GABA results are statistically indistinguishable for $\lambda_{\min} \in [0.02, 0.10]$ across the C-MAPSS subsets, so 0.05 is comfortably in the robust regime.

Interactive: Watch The Floor Engage

Drag the raw EMA slider from 0 to 1. The middle bars (clamped) light up amber when the floor is active. The right bars (renormalised) show the final $\lambda^*$. The bottom panel plots the input-output transfer curve so you can see the ‘knee’ at $\lambda_{\min}$ explicitly.

Try this. Drag raw λ_rul to 0. Without the floor, output would also be 0. With the paper's λ_min = 0.05, output is pinned at 0.04762 — the analytic lower bound. Now move λ_min to 0 and watch the bottom curve become y = x: no anti-windup, no bound, full task suppression possible. Move λ_min to 0.5 and the curve collapses to a horizontal line at 0.5: the floor is so aggressive that GABA degenerates to uniform weighting, defeating the closed form.

Python: Three Scenarios From Scratch

Implement floor_and_renorm in pure NumPy and exercise it on three representative inputs: paper-realistic 500× imbalance, pathological near-zero, and balanced no-op. The third confirms the graceful-degradation property: the floor does nothing when nothing needs fixing.

A. raw   = [0.002 0.998]
   clamp = [0.05  0.998]
   out   = [0.047710, 0.952290]   sum = 1.000000

B. raw   = [1.00e-08, 1.000000]
   clamp = [0.05       0.99999999]
   out   = [0.047619, 0.952381]

C. raw   = [0.5 0.5]
   clamp = [0.5 0.5]
   out   = [0.500000, 0.500000]
   unchanged? True

K=2 bounds:  lower = lam_min / (1 + lam_min)   = 0.047619
             upper = 1     / (1 + lam_min)     = 0.952381
             paper claim: in [lam_min, 1-lam_min] = [0.05, 0.95]  (approx)

1"""Floor + renormalisation: paper equation 6 from scratch."""
2
3import numpy as np
4
5
6def floor_and_renorm(ema: np.ndarray, lam_min: float) -> np.ndarray:
7    """Paper eq. 6: clamp at floor, then renormalise to the simplex.
8
9        lambda_i* = max(ema_i, lam_min) / sum_j max(ema_j, lam_min)
10
11    Guarantees lambda_i* in [lam_min/(1+lam_min*(K-1)), 1/(1+lam_min*(K-1))]
12    for K tasks; for K=2 this is approximately [lam_min, 1-lam_min].
13    """
14    clamped = np.maximum(ema, lam_min)
15    return clamped / clamped.sum()
16
17
18# ---------- Paper canonical floor ----------
19lam_min = 0.05
20
21
22# ---------- Scenario A: paper-realistic 500x imbalance ----------
23ema_a = np.array([0.002, 0.998])
24out_a = floor_and_renorm(ema_a, lam_min)
25print(f"A. raw   = {ema_a}")
26print(f"   clamp = {np.maximum(ema_a, lam_min)}")
27print(f"   out   = [{out_a[0]:.6f}, {out_a[1]:.6f}]   sum = {out_a.sum():.6f}")
28
29
30# ---------- Scenario B: extreme imbalance (g_health -> 0) ----------
31ema_b = np.array([1e-8, 1.0 - 1e-8])
32out_b = floor_and_renorm(ema_b, lam_min)
33print(f"\nB. raw   = [{ema_b[0]:.2e}, {ema_b[1]:.6f}]")
34print(f"   clamp = {np.maximum(ema_b, lam_min)}")
35print(f"   out   = [{out_b[0]:.6f}, {out_b[1]:.6f}]")
36
37
38# ---------- Scenario C: balanced (no clamping needed) ----------
39ema_c = np.array([0.5, 0.5])
40out_c = floor_and_renorm(ema_c, lam_min)
41print(f"\nC. raw   = {ema_c}")
42print(f"   clamp = {np.maximum(ema_c, lam_min)}")
43print(f"   out   = [{out_c[0]:.6f}, {out_c[1]:.6f}]")
44print(f"   unchanged? {np.allclose(ema_c, out_c)}")
45
46
47# ---------- Bounded-weight guarantee (K=2 closed form) ----------
48lower = lam_min / (1 + lam_min)
49upper = 1.0     / (1 + lam_min)
50print(f"\nK=2 bounds:  lower = lam_min / (1 + lam_min)   = {lower:.6f}")
51print(f"             upper = 1     / (1 + lam_min)     = {upper:.6f}")
52print(f"             paper claim: in [lam_min, 1-lam_min] = [0.05, 0.95]  (approx)")

PyTorch: clamp + renormalise (Paper Code)

The paper code in grace/core/gaba.py:134-135 is two lines: weights = ema_w.clamp(min=self.min_weight) followed by weights = weights / weights.sum(). We reproduce both verbatim and confirm that the boundary case $\hat{\lambda} = [0, 1]$ produces exactly the analytic bound $[0.04762, 0.95238]$.

A. raw = [0.002, 0.998] -> out = [0.04770992398262024, 0.9522900581359863]
B. raw = [1e-08, 0.9999999...] -> out = [0.047619... , 0.952381...]
C. raw = [0.5, 0.5] -> out = [0.5, 0.5]

at ema=[0, 1]: out = [0.04761904925107956, 0.9523810148239136]
theoretical bound: [0.047619, 0.952381]

grad through clamp at ema=[0.002, 0.998]:
  d(sum out) / d ema = [0.0, 0.0]
  (sum out is identically 1, so grad should be 0)

1"""Paper code: ema_w.clamp(min=lam_min) / sum (grace/core/gaba.py:134)."""
2
3import torch
4
5
6# ---------- Paper canonical hyperparameters ----------
7LAM_MIN = 0.05
8
9
10def gaba_floor_and_renorm(ema_w: torch.Tensor, min_weight: float = LAM_MIN) -> torch.Tensor:
11    """Floor + renormalise (paper grace/core/gaba.py lines 134-135).
12
13    weights = ema_w.clamp(min=self.min_weight)
14    weights = weights / weights.sum()
15    """
16    weights = ema_w.clamp(min=min_weight)
17    weights = weights / weights.sum()
18    return weights
19
20
21# ---------- Three test cases ----------
22ema_a = torch.tensor([0.002, 0.998])
23ema_b = torch.tensor([1e-8, 1.0 - 1e-8])
24ema_c = torch.tensor([0.5, 0.5])
25
26print(f"A. raw = {ema_a.tolist()} -> out = {gaba_floor_and_renorm(ema_a).tolist()}")
27print(f"B. raw = {ema_b.tolist()} -> out = {gaba_floor_and_renorm(ema_b).tolist()}")
28print(f"C. raw = {ema_c.tolist()} -> out = {gaba_floor_and_renorm(ema_c).tolist()}")
29
30
31# ---------- Confirm the bound at the saturated regime ----------
32ema_extreme = torch.tensor([0.0, 1.0])
33out_extreme = gaba_floor_and_renorm(ema_extreme)
34print(f"\nat ema=[0, 1]: out = {out_extreme.tolist()}")
35print(f"theoretical bound: [{LAM_MIN / (1 + LAM_MIN):.6f}, {1.0 / (1 + LAM_MIN):.6f}]")
36
37
38# ---------- Differentiability of the clamp (forward only) ----------
39ema_grad = torch.tensor([0.002, 0.998], requires_grad=True)
40out_grad = gaba_floor_and_renorm(ema_grad)
41out_grad.sum().backward()
42print(f"\ngrad through clamp at ema=[0.002, 0.998]:")
43print(f"  d(sum out) / d ema = {ema_grad.grad.tolist()}")
44print(f"  (sum out is identically 1, so grad should be 0)")

Floors In Other Fields

The same recipe — floor + renormalise — appears in many fields under many names. It is the canonical way to add stability to an unconstrained adaptive controller.

Pitfalls In The Floor + Renormalise Step

Takeaway

• Paper Eq. 6 is two lines. weights = ema_w.clamp(min=lam_min); weights = weights / weights.sum(). That is the entire stabiliser.
• The floor is anti-windup. Without it, a streak of bad batches can drive a task weight to ~0 and the EMA takes hundreds of steps to recover. With it, every task always carries at least $\lambda_{\min}/(1 + \lambda_{\min})$ weight.
• Bounded-weight guarantee. For K=2: $\lambda^* \in [0.04762, 0.95238]$ per step, deterministic. This is the property GradNorm cannot match.
• Paper picks $\lambda_{\min} = 0.05$. Robust across $[0.02, 0.10]$ per the §5.8 ablation. Small enough not to distort the closed form; large enough to provide real anti-windup.
• Graceful no-op when not needed. If all $\hat{\lambda}_i \geq \lambda_{\min}$, floor is identity and renormalisation divides by 1 — the entire stabiliser becomes invisible.
• Same recipe everywhere. Gradient clipping, ε-greedy floors, label smoothing, PID anti-windup, audio gating, portfolio diversification: all instances of ‘floor + renormalise to add stability to an adaptive controller’.