EMA as a First-Order IIR Low-Pass Filter

Hook: A Microphone, A Hiss, And A Capacitor

Plug a vocal microphone into a cheap mixing board and you hear two things: the singer's voice (the slow signal you want) and an air hiss from the preamp (the fast noise you don't). To clean it up, every audio engineer reaches for the same tool: a low-pass filter. Pass the slow voice through; reject the high-frequency hiss. The simplest realisation is a single resistor and a single capacitor — a textbook RC low-pass network — and the cutoff frequency is set by $f_c = 1 / (2 \pi R C)$. That circuit dates to 1899 and it has not gone out of style.

GABA solves the same problem in software. The closed-form rule from §19.1 emits a per-step weight that is brutally noisy: the per-batch gradient ratio in the paper fluctuates between 10× and 10,000× from one minibatch to the next (paper main.tex:576), so the un-smoothed $\lambda_i^{*}$ swings wildly. Feeding that directly into the optimiser would oscillate the loss combination and wreck training. The fix is exactly the audio-engineer move: put a low-pass filter between the controller and the actuator. Paper main.tex:387 names that filter explicitly — “The EMA with $\beta = 0.99$ serves as a first-order IIR low-pass filter (time constant $\sim 100$ steps) that smooths stochastic gradient noise, preventing oscillation.”

Why this section matters. The EMA buffer is not just an empirical smoother — it is a digital realisation of the same RC low-pass topology that an audio engineer reaches for. Once we identify the filter, we get its frequency response, its time constant, and its noise-rejection figure for free from a hundred years of DSP textbooks. We'll prove the time constant is exactly $\tau = 1/(1-\beta) = 100$ samples, derive the magnitude response, and show why $\beta = 0.99$ is the lowest gain that still rejects the paper's observed batch-to-batch noise by 30+ dB.

What Is A First-Order IIR Low-Pass Filter?

A digital filter is a recipe that turns one sequence of samples $x[t]$ into another sequence $y[t]$. There are exactly two flavours.

IIR filters are recursive: the new output depends on past outputs. That recursion is what gives them infinite memory and an exponential decay structure. The simplest non-trivial IIR is the first-order low-pass:

$y[t] = \beta \cdot y[t-1] + (1 - \beta) \cdot x[t], \qquad 0 \le \beta < 1.$

Two coefficients, both fixed by a single parameter $\beta$. The new output is a convex combination of yesterday's output (weight $\beta$) and today's input (weight $1-\beta$). Because the coefficients sum to 1, the filter passes a constant signal through unchanged — what engineers call “unity DC gain.”

The transfer function (z-transform of the recursion) is

$H(z) = \frac{1 - \beta}{1 - \beta z^{-1}}, \qquad \text{single pole at } z = \beta.$

That single pole on the real axis at $z = \beta$ is what makes this a one-pole low-pass. The pole at $z = 0.99$ sits very close to the unit circle $|z| = 1$ — very close to the boundary of instability — which is exactly why it has such a long memory.

Why The EMA Update IS That Filter

Compare the two recursions side by side. The paper's EMA update is paper eq. 5 (main.tex:346):

$\hat{\lambda}_i^{(t)} = \beta \cdot \hat{\lambda}_i^{(t-1)} + (1 - \beta) \cdot \lambda_i^{(t)}, \qquad \beta = 0.99.$

The first-order IIR low-pass recursion is

$y[t] = \beta \cdot y[t-1] + (1 - \beta) \cdot x[t].$

Rename $x[t] = \lambda_i^{(t)}$ (the raw per-step weight from the closed-form P-controller) and $y[t] = \hat{\lambda}_i^{(t)}$ (the smoothed weight). The two recursions are the same equation, character for character. The EMA buffer is a first-order digital low-pass IIR filter applied to the closed-form weight stream.

The impulse response — what the filter does to a single, isolated sample at $t = 0$ — falls out of the recursion by unrolling:

$h[k] = (1 - \beta) \cdot \beta^{k}, \qquad k = 0, 1, 2, \ldots$

Geometric decay with ratio $\beta$. At $\beta = 0.99$, the weight on a sample 100 steps ago is $(0.01) \cdot (0.99)^{100} \approx 0.0037$ — about $1/e$ of the weight on the most recent sample, but spread across all future outputs. This unbounded but exponentially-decaying memory is the “I” in IIR.

The Time Constant: τ = 1 / (1 − β) = 100 Steps

Every first-order low-pass has a single number that summarises its memory: the time constant $\tau$. For the discrete EMA filter it is

$\tau = \frac{1}{1 - \beta} \quad \text{samples}.$

The paper's default $\beta = 0.99$ gives $\tau = 100$. Three concrete consequences fall out of this number:

• Step-response settling. If the input jumps from 0 to 1 at $t = 0$, the filter output reaches $1 - e^{-1} \approx 0.632$ at $t = \tau$, $1 - e^{-2} \approx 0.865$ at $t = 2\tau$, and $1 - e^{-3} \approx 0.950$ at $t = 3\tau$. Same canonical milestones as an analog RC low-pass.
• Effective averaging window. The geometric weights $(1-\beta)\beta^k$ have an effective window of about $\tau$ samples — ~63% of the total weight comes from the most recent $\tau$ samples. Equivalent to a uniform moving average of length $\tau$, but causal and recursive (constant memory and compute per step instead of $O(\tau)$).
• Adaptation speed. If the ‘true’ $\lambda_i$ jumps from 0.5 to 0.998 (the paper's post-warmup transition), the filter takes about $3\tau = 300$ steps to converge to within 5% of the new value. Paper main.tex:576 confirms this empirically: “$\lambda_{\text{health}}$ converges from the equal initialization (0.5) to $> 0.99$ within ~10 epochs” — and 10 epochs at the paper's ~30 batches/epoch is $\sim 300$ steps, exactly $3\tau$.

Frequency Response And The −3 dB Cutoff

Substitute $z = e^{j\omega}$ into the transfer function to get the steady-state response to a sinusoid at angular frequency $\omega$ rad/sample:

$|H(e^{j\omega})| = \frac{1 - \beta}{\sqrt{1 - 2\beta \cos\omega + \beta^2}}.$

At $\omega = 0$ (DC): $|H| = (1-\beta)/(1-\beta) = 1$ — unity gain at zero frequency. At $\omega = \pi$ (Nyquist): $|H| = (1-\beta)/(1+\beta)$ — the maximum possible attenuation. For $\beta = 0.99$, that is $0.01/1.99 \approx 5 \times 10^{-3}$ or $\approx -46$ dB. Between DC and Nyquist the magnitude is monotonically decreasing.

The conventional summary number is the −3 dB cutoff $f_c$ — the frequency at which the magnitude drops to $1/\sqrt{2} \approx 0.707$. Solving for the first-order IIR gives

$\cos\omega_c = 1 - \frac{(1-\beta)^2}{2\beta} \approx 1 - \frac{1}{2\tau^2}, \qquad \omega_c \approx \frac{1}{\tau} \;\text{(for } \beta \to 1\text{)}.$

For $\beta = 0.99$: $\omega_c \approx 0.01$ rad/sample, so $f_c \approx 0.01/(2\pi) \approx 1.6 \times 10^{-3}$ cycles/step. Past that cutoff the filter rolls off at the textbook −20 dB/decade for any first-order low-pass — the same Bode slope you see on a single-RC stage.

Interactive: Impulse And Frequency Response

Two coordinated panels. The left shows the impulse-response weights $h[k] = (1-\beta) \beta^k$ — the memory of the filter, stem-plot style. The right shows the magnitude Bode plot $|H(e^{j\omega})|$ in dB versus normalised frequency. Move the $\beta$ slider to retune the filter; move the probe-frequency slider to inject a sinusoidal disturbance and read off the steady-state attenuation.

Three guided experiments to run on the viz:

• Memory length. Click the $\beta = 0.99$ preset. Read the time-constant marker at $\tau = 100$ on the impulse panel. About 63% of the total weight sits to the left of that marker — this is the “effective averaging window.”
• Noise rejection at paper-realistic frequencies. Set the probe slider to $f = 0.1$ cycles/step (one oscillation every 10 steps, paper-realistic for batch-to-batch gradient noise). Read the dB number on the right panel: at $\beta = 0.99$ the attenuation is ~−36 dB, i.e. the noise amplitude shrinks by a factor of ~63.
• Why $\beta$ can't be smaller. Drop $\beta$ to 0.5. The time constant collapses to 2 steps, the cutoff frequency moves right by a factor of 50, and the same probe at $f = 0.1$ now sees only ~−3 dB of attenuation. The smoothed weight would shadow the noisy raw weight almost step for step — precisely the oscillation regime that paper main.tex:553 attributes to GradNorm's divergence on 1/5 N-CMAPSS seeds.

Why β = 0.99 Survives The 10× to 10,000× Disturbance Range

The paper's Fig. gradient_dynamics reports that the per-batch gradient ratio $\|g_{\text{rul}}\| / \|g_{\text{health}}\|$ fluctuates between 10× and 10,000× on the log scale, thin-line per-seed curves piling on top of a slowly-drifting mean (paper main.tex:576). On the simplex this corresponds to a raw un-smoothed $\lambda^{*}$ swinging between approximately 0.0001 and 0.1 every few batches.

The $\beta = 0.99$ row sits at the knee: enough rejection (63×) that the per-batch noise is invisible in the smoothed weight, but not so much that the filter drags behind a real regime shift. The paper main.tex:647 reports that across 40 runs the converged smoothed weight is $\lambda_{\text{health}} = 0.995 \pm 0.001$ — a coefficient of variation of 0.14% — which is only achievable with this much rejection.

Python: IIR Filter From Scratch (NumPy)

Build the filter as a small class with three methods: a single sample step, a vector wrapper, and a frequency-domain probe. Run a unit-step input to observe the canonical 63%/86%/95% milestones at$\tau, 2\tau, 3\tau$, and a sinusoidal probe to verify the magnitude formula numerically.

y[ 0 ] = 0.0100
y[100] = 0.6340  (~63%)
y[200] = 0.8660  (~86%)
y[300] = 0.9510  (~95%)

|H| at f=0.05 = 0.03191  (-29.9 dB)
attenuation = 31x

1"""GABA's EMA buffer as a first-order IIR low-pass filter.
2
3Pure NumPy. Runs the SAME numerical recursion as paper grace/core/gaba.py
4but documents it in DSP notation (impulse response, frequency response,
5time constant, -3 dB cutoff) so the connection to classical signal
6processing is explicit.
7
8Filter law (paper eq. 5):
9
10    y[t] = beta * y[t-1] + (1 - beta) * x[t]
11
12Transfer function (z-domain):
13
14    H(z) = (1 - beta) / (1 - beta * z^{-1})        # one pole at z = beta
15
16Impulse response:
17
18    h[k] = (1 - beta) * beta^k    for k >= 0
19"""
20
21import numpy as np
22
23
24class FirstOrderIIRFilter:
25    """Identical numerics to GABA's EMA buffer."""
26
27    def __init__(self, beta=0.99, init=0.0):
28        self.beta = beta
29        self.y = float(init)             # filter state (one float per channel)
30        self.tau = 1.0 / (1.0 - beta)    # time constant in samples
31
32    def step(self, x):
33        """One sample in, one sample out (paper eq. 5)."""
34        self.y = self.beta * self.y + (1.0 - self.beta) * x
35        return self.y
36
37    def filter(self, signal):
38        """Vector form: filter a length-N input array."""
39        out = np.empty_like(signal, dtype=np.float64)
40        for t, x in enumerate(signal):
41            out[t] = self.step(x)
42        return out
43
44    def magnitude_response(self, omega):
45        """|H(e^{j omega})| for a normalised angular frequency omega."""
46        b = self.beta
47        return (1.0 - b) / np.sqrt(1.0 - 2.0 * b * np.cos(omega) + b * b)
48
49
50# ---- Step-response demo: x jumps from 0 to 1 at t = 0 ----
51filt = FirstOrderIIRFilter(beta=0.99)
52
53x = np.ones(400)                     # unit step input
54y = filt.filter(x)
55
56# Settling milestones for tau = 100 steps:
57print(f"y[ 0 ] = {y[0]:.4f}")        # one sample after the step
58print(f"y[100] = {y[100]:.4f}  (~63%)")
59print(f"y[200] = {y[200]:.4f}  (~86%)")
60print(f"y[300] = {y[300]:.4f}  (~95%)")
61
62# ---- Sinusoidal disturbance demo at f = 0.05 cycles/step (paper-realistic) ----
63freq = 0.05
64omega = 2 * np.pi * freq
65mag = filt.magnitude_response(omega)
66print(f"\n|H| at f={freq} = {mag:.5f}  ({20 * np.log10(mag):.1f} dB)")
67print(f"attenuation = {1 / mag:.0f}x")

The output is the textbook step response: 1.00% absorbed in one sample, 63.4% at $t = \tau = 100$, 86.6% at $2\tau$, 95.1% at $3\tau$. The frequency probe at $f = 0.05$ hits −29.9 dB — a 31× attenuation on a paper-realistic disturbance frequency.

PyTorch: The EMA Branch Of GABALoss Verbatim

The production code in grace/core/gaba.py writes the EMA recursion directly inside GABALoss.forward_k. To isolate it as a standalone module, we extract just the recursion and the buffer-registration logic. The result is bit-exact to the EMA branch of GABALoss when the floor is disabled.

tau (samples) = 100.0
step   1: lambda_hat = (0.500000, 0.500000)
step 100: lambda_hat = (0.500000, 0.500000)
step 101: lambda_hat = (0.495020, 0.504980)
step 150: lambda_hat = (0.198047, 0.801953)
step 200: lambda_hat = (0.183580, 0.816420)

1"""The EMA branch of paper grace/core/gaba.py:GABALoss.forward_k.
2
3Same recursion as the NumPy class above, but on PyTorch tensors using
4register_buffer so the state moves with .to(device) and survives
5checkpoint save/load.
6"""
7
8import torch
9import torch.nn as nn
10
11
12class GabaEMABuffer(nn.Module):
13    """The EMA stabiliser of GABA — paper eq. 5.
14
15    Holds the smoothed weight vector lambda_hat as a (K,) buffer.
16    No learnable parameters. Bit-exact to the EMA branch of paper
17    GABALoss.forward_k when min_weight = 0.
18    """
19
20    def __init__(self, beta=0.99, n_tasks=2):
21        super().__init__()
22        self.beta = beta
23        self.n_tasks = n_tasks
24        # Initialise at uniform 1/K (paper Algorithm 1 line 2).
25        self.register_buffer("lambda_hat", torch.full((n_tasks,), 1.0 / n_tasks))
26
27    @property
28    def tau(self):
29        """Time constant in samples (1 / (1 - beta))."""
30        return 1.0 / (1.0 - self.beta)
31
32    def forward(self, raw):
33        """One filter step. raw is the (K,) closed-form weight from sec. 19.1."""
34        # In-place would silently break autograd if raw is grad-tracked.
35        new_state = self.beta * self.lambda_hat + (1.0 - self.beta) * raw.detach()
36        # detach() is intentional: the EMA state is OPTIMISED state, not a
37        # quantity we backprop through. Same convention as paper grace/core/gaba.py.
38        self.lambda_hat = new_state.detach()
39        return self.lambda_hat
40
41
42# ---- Smoke test: 200-step trajectory tracking the paper's 0.5 -> 0.998 jump ----
43torch.manual_seed(0)
44
45ema = GabaEMABuffer(beta=0.99, n_tasks=2)
46print(f"tau (samples) = {ema.tau}")
47
48# Steps 1-100: warmup-style raw input still 0.5
49# Steps 101-200: imbalanced regime — closed-form rule outputs (0.002, 0.998)
50trajectory = []
51for t in range(1, 201):
52    if t <= 100:
53        raw = torch.tensor([0.5, 0.5])
54    else:
55        raw = torch.tensor([0.002, 0.998])
56    out = ema(raw)
57    trajectory.append((t, out[0].item(), out[1].item()))
58
59# Print key milestones
60for t in [1, 100, 101, 150, 200]:
61    _, lr, lh = trajectory[t - 1]
62    print(f"step {t:3d}: lambda_hat = ({lr:.6f}, {lh:.6f})")

The smoke test reproduces the paper's warmup-to-active transition: 100 steps of equal-share input leave the buffer untouched (it was already at 0.5); the next 100 steps push it from 0.5 toward 0.998 with the canonical exponential trajectory, hitting about 0.184/0.816 at $t = \tau = 100$ post-transition — the textbook 63.2% settling.

The Same Filter In Other Domains

The first-order IIR low-pass is one of the most widely-deployed algorithms in engineering. It shows up under many names because the recursion is the same regardless of what is being filtered.

The pattern is “recursion + state buffer + DC-unity gain.” The specific value of $\beta$ in each row encodes a domain-specific tradeoff between adaptation speed and noise rejection — but the algorithm and the analysis are identical. Recognising the filter in one row gives you the time constant, the cutoff frequency, and the noise-rejection figure in every other row for free.

Cross-domain analogy: BatchNorm running stats. BatchNorm tracks the running mean and variance of activations with an EMA at $\beta = 0.99$ — identical numerics to GABA's weight smoother, identical time constant, identical motivation (per-batch statistics are noisy and you want the steady trend). When you train a CNN on ImageNet you rely on this filter every step without thinking about it. GABA is applying exactly the same trick to a different signal: per-batch gradient ratios instead of per-batch activation statistics.

Pitfalls In Tuning Or Replacing The EMA

Takeaway

• The EMA recursion IS a first-order IIR low-pass filter, character for character. Paper eq. 5 and the textbook DSP recursion are the same equation; the EMA buffer is a digital realisation of the same RC low-pass topology audio engineers have been using since 1899.
• Time constant $\tau = 1/(1-\beta) = 100$ samples. Step response hits 63%/86%/95% at $\tau, 2\tau, 3\tau$. Effective averaging window is ~τ samples.
• Cutoff frequency $f_c \approx 1.6 \times 10^{-3}$ cycles/step. Magnitude rolls off at −20 dB/decade past the cutoff. At paper-realistic disturbance bandwidths (~0.05–0.1 cycles/step) the attenuation is 30+ dB — comfortably above the engineering rule of thumb.
• The default $\beta = 0.99$ is the lowest value satisfying the 30 dB rule on the paper's observed disturbance bandwidth. Going higher slows adaptation unnecessarily; going lower fails the rule and re-introduces the oscillation regime.
• The same filter is everywhere. Adam moments, BatchNorm running stats, BYOL/MoCo targets, Polyak averaging, RL target networks, RC analog audio filters — the recursion, the time constant, and the Bode plot are identical. Recognising the EMA in GABA gives you all of those analyses for free.
• Coming next. §19.3 identifies the floor $\lambda_{\min} = 0.05$ as the anti-windup mechanism that completes the controller, and proves the bounded-weight guarantee $\lambda^{*} \in [\lambda_{\min}, 1 - \lambda_{\min}]$ that paper main.tex:387 calls ‘a stability property absent from loss-based approaches.’