The Core Discovery: Equal Weighting

Learning Objectives

By the end of this section, you will:

1. Understand the core discovery that equal task weights are optimal
2. Examine the experimental evidence across all C-MAPSS datasets
3. Explain why equal weighting works for RUL prediction
4. Recognize the importance of loss normalization
5. Appreciate how this discovery led to AMNL

Why This Matters: The discovery that equal task weights (0.5/0.5) consistently outperform adaptive methods was unexpected and counterintuitive. It challenged the prevailing wisdom that sophisticated weight adaptation is always better. This discovery is the foundation of AMNL and the key to achieving state-of-the-art performance.

The Discovery

Our research began with a comprehensive evaluation of multi-task weighting methods. What we found was surprising.

The Hypothesis

We initially hypothesized that adaptive methods (uncertainty weighting, GradNorm, DWA) would outperform fixed weights because:

• They adapt to changing loss scales during training
• They balance gradient magnitudes automatically
• They are "principled" (derived from optimization theory)

The Surprise

Extensive experiments revealed the opposite:

1Hypothesis: Adaptive > Fixed
2
3Reality:
4  Fixed (0.5/0.5) > Adaptive methods
5
6Specifically:
7  Equal weights (0.5/0.5) consistently achieved the best results
8  across ALL four C-MAPSS datasets.

Experimental Evidence

We conducted systematic weight sweep experiments across all datasets.

Weight Sweep Results

Testing all weight combinations $(\lambda_{\text{RUL}}, \lambda_{\text{health}}) = (\lambda, 1-\lambda)$:

Optimal Weight Location

1Performance vs. λ_RUL:
2
3RMSE
4 17 ─┐                                    ╭─
5    │╲                                  ╱
6 16 ─┤ ╲                              ╱
7    │  ╲                            ╱
8 15 ─┤   ╲                        ╱
9    │    ╲                      ╱
10 14 ─┤     ╲                  ╱
11    │      ╲                ╱
12 13 ─┤       ╲    ╱╲      ╱
13    │        ╲__╱  ╲____╱
14 12 ─┤                ▼
15    │            λ = 0.5 (optimal)
16 11 ─┴──┬──┬──┬──┬──┬──┬──┬──┬──
17       0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
18                     λ_RUL

Statistical Significance

We verified the results are statistically significant:

Why Equal Weights Work

Several factors explain why equal weighting is optimal for RUL prediction.

Factor 1: Maximum Regularization

Factor 2: Gradient Balance

With proper loss normalization, equal weights produce balanced gradients:

Where $\tilde{\mathcal{L}}$ denotes normalized losses. Neither task dominates gradient updates.

Factor 3: Complementary Information

The two tasks provide complementary supervision:

Equal weighting ensures both types of information contribute equally to learning.

The Key: Loss Normalization

Equal weighting only works with proper loss normalization.

The Problem Without Normalization

Raw losses have vastly different scales:

1Without normalization:
2  L_RUL ≈ 100-2000  (MSE on cycles)
3  L_health ≈ 0.5-2  (cross-entropy)
4
5With λ = 0.5 for both:
6  0.5 × 1000 + 0.5 × 1.5 = 500.75
7
8RUL contribution: 500 / 500.75 = 99.85%
9Health contribution: 0.15%
10
11→ Equal weights ≠ equal contribution!

The Solution: Normalize First

AMNL normalizes losses before weighting:

Where EMA is exponential moving average for stable normalization.

1With normalization:
2  L̃_RUL = 1000 / 1000 = 1.0
3  L̃_health = 1.5 / 1.5 = 1.0
4
5With λ = 0.5 for both:
6  0.5 × 1.0 + 0.5 × 1.0 = 1.0
7
8RUL contribution: 50%
9Health contribution: 50%
10
11→ Equal weights = equal contribution!

Summary

In this section, we presented the core discovery behind AMNL:

1. Discovery: Equal weights (0.5/0.5) are optimal for RUL + health
2. Evidence: Consistent across all C-MAPSS datasets
3. Why it works: Maximum regularization, gradient balance
4. Critical requirement: Loss normalization
5. AMNL formula: Normalized losses + equal weights

Looking Ahead: We have established that equal weighting works when combined with normalization. The next section presents the complete mathematical formulation of AMNL, including the normalization mechanism and the full loss equation.

With the discovery explained, we now formalize AMNL mathematically.