Why 0.5/0.5 Provides Superior Regularization

Learning Objectives

By the end of this section, you will:

1. Understand the regularization theory behind equal weighting
2. Analyze gradient balance under normalized losses
3. Apply information-theoretic reasoning to task weighting
4. Interpret empirical evidence supporting the 0.5/0.5 split
5. Explain why unequal weights hurt performance in the RUL + health setting

Why This Matters: The discovery that equal weights are optimal is not arbitrary—it emerges from fundamental principles of regularization, gradient dynamics, and information theory. Understanding these principles explains why AMNL succeeds and provides confidence that the approach generalizes beyond C-MAPSS.

Regularization Theory

The health classification task acts as a regularizer for RUL prediction. Equal weighting maximizes this regularization effect.

Multi-Task Learning as Regularization

In multi-task learning, auxiliary tasks constrain the hypothesis space:

Where:

• $\mathcal{H}_{\text{RUL}}$: Functions that fit RUL data well
• $\mathcal{H}_{\text{health}}$: Functions that fit health data well
• $\mathcal{H}_{\text{MTL}}$: Intersection (smaller, more regularized)

The auxiliary task shrinks the hypothesis space, reducing overfitting.

Weight Impact on Regularization Strength

Consider the regularization strength as a function of the health weight $\lambda_{\text{health}}$:

Formal Regularization Bound

From a bias-variance perspective, the expected error can be decomposed:

With multi-task learning:

• Bias: Slightly increased (must satisfy both tasks)
• Variance: Significantly reduced (constrained hypothesis space)
• Net effect: Lower total error when tasks are related

Gradient Balance Analysis

Equal weights combined with normalization produce balanced gradient contributions.

Gradient Flow in AMNL

The total gradient under AMNL:

Since $\mathcal{L}_i / \mu_i \approx 1$ after normalization, we expect:

Why Balance Matters

Gradient Interference

When tasks share parameters, their gradients can interfere:

• $\cos(\theta) > 0$: Positive transfer (gradients aligned)
• $\cos(\theta) < 0$: Negative transfer (gradients conflict)
• $\cos(\theta) \approx 0$: Independent tasks

For RUL and health, we observe:

Information-Theoretic View

Information theory provides another lens to understand equal weighting.

Mutual Information Perspective

The encoder learns a representation Z that should maximize:

Subject to the constraint that Z is computed from input X:

Equal weighting maximizes the joint information extraction from Z.

Label Entropy Analysis

Compare the entropy of each task's labels:

Information Bottleneck Principle

The Information Bottleneck (IB) framework suggests the optimal representation Z minimizes:

For multi-task learning, this extends to:

When $\beta_1 = \beta_2$, the representation must capture information relevant to both tasks equally, leading to more generalizable features.

Empirical Validation

Theory predicts 0.5/0.5 is optimal. Experiments confirm this across all datasets.

Weight Sensitivity Analysis

Fine-grained weight sweep results (RMSE):

The optimal is consistently at or very near λ = 0.5.

Performance Degradation Analysis

How much does performance degrade as we deviate from 0.5?

1Deviation from λ = 0.5 vs. RMSE increase (average across datasets):
2
3Δλ    | RMSE Increase | Relative Degradation
4------+---------------+---------------------
5±0.00 |     0.00      | Baseline (optimal)
6±0.05 |     0.45      | +3.4%
7±0.10 |     1.10      | +8.3%
8±0.15 |     1.85      | +13.9%
9±0.20 |     2.70      | +20.3%
10±0.30 |     4.30      | +32.3%
11±0.40 |     6.20      | +46.6%

Performance degrades roughly quadratically as we deviate from the optimal.

Ablation: Normalization Effect

Does equal weighting work without normalization?

Cross-Architecture Validation

Does 0.5/0.5 generalize across different encoder architectures?

The optimal weight is consistently 0.5 (or within 0.05) across all encoder architectures tested.

Summary

In this section, we established why 0.5/0.5 provides superior regularization:

1. Regularization theory: Maximizes $\lambda(1-\lambda)$ at λ = 0.5
2. Gradient balance: Equal contribution from both tasks
3. Information theory: Balanced extraction of complementary information
4. Empirical validation: Consistent across datasets and architectures
5. Normalization essential: Equal weights only work with proper normalization

Looking Ahead: With the theoretical foundation established, the final section presents the complete PyTorch implementation of AMNL—production-ready code that you can use directly in your own predictive maintenance projects.

Having understood why equal weighting works, we now implement AMNL in code.