The Loss-Combination Problem

Two Voices in a Choir

In a duet, the two voices are not interchangeable. A mezzo-soprano and a baritone have different ranges, different volumes, different tonal weights. A naive sound engineer who slides both faders to the same level produces something the mezzo dominates — or, with the wrong room, a baritone that buries her. The job of mixing is to WEIGHT the channels so the result sounds balanced.

Multi-task learning has the same problem. The RUL regression loss is on the scale of squared cycles (tens to hundreds); the health classification loss is on the scale of cross-entropy (zero to a couple). Add them with equal weight and the regression dominates the optimisation. Picking the weights well — statically, dynamically, adaptively — is the central engineering problem of every paper in this corner of the literature.

The Weighted-Sum Formulation

The dominant approach is a single scalar loss formed as a weighted sum of per-task losses:

$\mathcal{L}_{\text{total}}(\theta) \;=\; \sum_{k=1}^{K} \lambda_k \, \mathcal{L}_k(\theta), \qquad \lambda_k \ge 0.$

Three properties of this formulation matter. (1) Smooth and differentiable: gradients flow through the sum into every shared parameter. (2) One scalar to optimise: standard back-prop and optimisers work unchanged. (3) Pareto-aware: every choice of $\lambda$ selects a point on the Pareto frontier of the per-task losses — a tradeoff curve where reducing one task's loss requires increasing another's.

Interactive: The Pareto Frontier

The diagram below sketches a toy Pareto frontier between two synthetic losses. Slide $\lambda$ from 0 to 1 and watch the optimum move along the curve. Notice the curve is non-linear: at the extreme ends, gaining a tiny bit on the favoured task costs a lot on the other.

Real C-MAPSS frontiers look qualitatively similar but with much wider scale gaps; one of the reasons GABA (Section 17) outperforms a simple $\lambda$ sweep is that it moves the operating point during training instead of committing to one upfront.

Five Ways to Choose the Weights

The first two are static; the next three update during training. The static ones are easy to reason about and reproducible; the dynamic ones tend to outperform them because the right $\lambda$ can change over the course of an epoch (early epochs benefit from one balance, late epochs from another).

Python: One Loss, Five Weighting Strategies

1import numpy as np
2
3# Synthetic per-task losses for one batch
4L_rul    = 12.5    # large because RUL targets span 0..125 cycles
5L_health = 0.8     # small because cross-entropy is bounded by ~log(3)
6
7# Five weighting schemes you will see across this book ----------------
8# 1. Fixed equal
9J_fixed_equal   = 0.5 * L_rul + 0.5 * L_health
10
11# 2. Fixed unequal (e.g., the 0.5/0.5 of AMNL but with the bigger task halved)
12J_fixed_quarter = 0.25 * L_rul + 0.75 * L_health
13
14# 3. Inverse-magnitude (rough first attempt at "equalising contributions")
15weights_inv  = np.array([1.0 / L_rul, 1.0 / L_health])
16weights_inv /= weights_inv.sum()
17J_inv = weights_inv[0] * L_rul + weights_inv[1] * L_health
18
19# 4. Uncertainty-weighted (Kendall, Gal, Cipolla 2018)
20log_sigma_rul    = 1.5     # learnable; we fix for this demo
21log_sigma_health = 0.3
22sig2_rul    = np.exp(2 * log_sigma_rul)
23sig2_health = np.exp(2 * log_sigma_health)
24J_unc = (L_rul / (2 * sig2_rul) + log_sigma_rul +
25         L_health / (2 * sig2_health) + log_sigma_health)
26
27# 5. GABA — adaptive, gradient-magnitude-driven (Section 17)
28# Placeholder; the real version measures gradient norms at each step
29lambda_rul, lambda_health = 0.005, 0.995    # GABA's typical converged values
30J_gaba = lambda_rul * L_rul + lambda_health * L_health
31
32print(f"fixed equal     J = {J_fixed_equal:.3f}")
33print(f"fixed 25/75     J = {J_fixed_quarter:.3f}")
34print(f"inverse mag     J = {J_inv:.3f}")
35print(f"uncertainty     J = {J_unc:.3f}")
36print(f"GABA            J = {J_gaba:.3f}")
37
38# fixed equal     J = 6.650
39# fixed 25/75     J = 3.725
40# inverse mag     J = 1.504
41# uncertainty     J = 1.310
42# GABA            J = 0.859

PyTorch: Combining Losses in the Training Loop

1import torch
2import torch.nn as nn
3import torch.nn.functional as F
4
5class DualTaskMLP(nn.Module):
6    def __init__(self):
7        super().__init__()
8        self.shared = nn.Sequential(nn.Linear(17, 32), nn.ReLU())
9        self.head_rul    = nn.Linear(32, 1)
10        self.head_health = nn.Linear(32, 3)
11    def forward(self, x):
12        h = self.shared(x)
13        return self.head_rul(h).squeeze(-1), self.head_health(h)
14
15
16# ----- Training step with combined loss -----
17torch.manual_seed(0)
18model = DualTaskMLP()
19opt   = torch.optim.AdamW(model.parameters(), lr=1e-3)
20
21x         = torch.randn(64, 17)
22y_rul     = torch.randn(64) * 30 + 60                 # synthetic RUL targets
23y_health  = torch.randint(0, 3, (64,))                # synthetic health labels
24LAMBDA    = 0.5                                        # task weight (static)
25
26for step in range(3):
27    opt.zero_grad()
28
29    pred_rul, pred_health = model(x)
30
31    L_rul    = F.mse_loss(pred_rul,    y_rul)
32    L_health = F.cross_entropy(pred_health, y_health)
33    L_total  = LAMBDA * L_rul + (1 - LAMBDA) * L_health
34
35    L_total.backward()
36    opt.step()
37
38    print(f"step {step}: L_rul={L_rul.item():.3f}  L_health={L_health.item():.3f}  L_total={L_total.item():.3f}")
39# step 0: L_rul=4250.871  L_health=1.205  L_total=2126.038
40# step 1: L_rul=3873.015  L_health=1.198  L_total=1937.107
41# step 2: L_rul=3528.504  L_health=1.190  L_total=1764.847

The Same Tradeoff in Other Domains

Every row is one paper's worth of literature on how to set the weights. The patterns transfer; what changes is the magnitude of each loss and the criticality of each task.

Two Pitfalls

What the rest of the book is about. Choosing lambda well. Section 4.4 examines the gradients themselves; Sections 14, 17, 21 give three concrete weighting strategies (failure-biased weights, GABA, GRACE) and benchmark them.

Takeaway

• The combined loss is $\sum_k \lambda_k \mathcal{L}_k$. Smooth, differentiable, single-scalar. Standard optimisers work.
• Equal lambda is rarely balanced. Per-task losses live on different scales; equal lambda silently lets the larger-scale loss dominate.
• Five weighting strategies, one design choice. Fixed equal, fixed unequal, inverse-magnitude, uncertainty, adaptive. The book's contribution lives in the last bucket.
• The training loop pattern is universal. Forward, two losses, one combined loss, backward, step. lambda is the only knob you change between AMNL, GABA, GRACE.