Shared vs. Task-Specific Parameters

The Swiss Army Knife

A Swiss Army knife shares one handle across many tools. The handle is cheap, ergonomic, and never specialises — but it is the same handle whether you reach for the corkscrew or the screwdriver. Each tool has a small task-specific blade that does its own job. If you wanted a top-tier corkscrew you would buy a dedicated one; the Swiss-knife corkscrew is a pragmatic compromise. Multi-task neural networks make the same compromise.

The handle is the shared backbone — CNN + BiLSTM + attention in our case. The blades are the task-specific heads — one for RUL regression, one for health classification. The handle is forced to serve both tasks; the blades are free to specialise. Whether the trade pays off depends on how related the tasks are and how the gradients are balanced — the central topic of the rest of this chapter.

Splitting the Parameter Set

Formally, partition the model's parameters into $\theta = (\theta_s, \theta_1, \ldots, \theta_K)$ where $\theta_s$ are shared and $\theta_k$ belongs to task $k$ only. The forward pass is

$\hat{y}^{(k)} = g_k\bigl(s(\mathbf{x}; \theta_s);\, \theta_k\bigr).$

Read it left to right: input $\mathbf{x}$ goes through the shared encoder$s$; that result is consumed by every task-specific head $g_k$. The shared encoder runs ONCE per forward pass — the cost of the heads is marginal.

Which Parameters Get Which Gradients

The combined loss is $\mathcal{L} = \sum_k \lambda_k \mathcal{L}_k$. The gradient with respect to shared parameters is the SUM of all per-task contributions; the gradient with respect to task-specific parameters is JUST the contribution from that task:

$\nabla_{\theta_s} \mathcal{L} \;=\; \sum_{k=1}^{K} \lambda_k \, \nabla_{\theta_s} \mathcal{L}_k, \qquad \nabla_{\theta_k} \mathcal{L} \;=\; \lambda_k \, \nabla_{\theta_k} \mathcal{L}_k.$

Python: Counting Parameters by Group

Counting parameters is the simplest way to feel where the model lives. For the tiny $17 \to 32 \to (1, 3)$ dual-task MLP from Section 4.1:

1# Counting shared vs task-specific parameters in our DualTaskMLP
2shared_params = 17 * 32 + 32      # W_shared + b_shared
3rul_params    = 32 * 1  + 1       # W_rul    + b_rul
4health_params = 32 * 3  + 3       # W_health + b_health
5total         = shared_params + rul_params + health_params
6
7print(f"shared    : {shared_params:>4}  (sees both tasks' gradients)")
8print(f"RUL head  : {rul_params:>4}  (sees only RUL gradient)")
9print(f"Health    : {health_params:>4}  (sees only health gradient)")
10print(f"total     : {total:>4}")
11print(f"shared %  : {100 * shared_params / total:.1f}%")
12
13# shared    :  576  (sees both tasks' gradients)
14# RUL head  :   33  (sees only RUL gradient)
15# Health    :   99  (sees only health gradient)
16# total     :  708
17# shared %  : 81.4%

81% of the parameters are shared. In the full backbone ( Chapter 11's 3.5M-parameter model) the shared share is ~99% — the heads are tiny relative to the trunk. The shared parameters are where the optimisation pressure lives.

PyTorch: parameters() vs named_parameters()

PyTorch exposes two sibling iterators. parameters() yields just the tensors; named_parameters() yields (name, tensor) pairs. The latter is invaluable for debugging, weight inspection, and per-group optimiser configs.

1import torch
2from torch import nn
3
4class DualTaskMLP(nn.Module):
5    def __init__(self, in_dim=17, hidden=32, n_classes=3):
6        super().__init__()
7        self.shared      = nn.Sequential(nn.Linear(in_dim, hidden), nn.ReLU())
8        self.head_rul    = nn.Linear(hidden, 1)
9        self.head_health = nn.Linear(hidden, n_classes)
10
11model = DualTaskMLP()
12
13# ----- Inspect parameter groups by name -----
14for name, p in model.named_parameters():
15    print(f"{name:30s}  {tuple(p.shape)}  {p.numel()}")
16
17# shared.0.weight                 (32, 17)  544
18# shared.0.bias                   (32,)      32
19# head_rul.weight                 (1, 32)    32
20# head_rul.bias                   (1,)        1
21# head_health.weight              (3, 32)    96
22# head_health.bias                (3,)        3
23
24
25# ----- Group parameters for separate optimisers -----
26shared_params = list(model.shared.parameters())
27rul_params    = list(model.head_rul.parameters())
28health_params = list(model.head_health.parameters())
29
30opt = torch.optim.AdamW([
31    {"params": shared_params, "lr": 1e-3, "weight_decay": 1e-4},
32    {"params": rul_params,    "lr": 1e-3, "weight_decay": 1e-4},
33    {"params": health_params, "lr": 1e-3, "weight_decay": 1e-4},
34])
35
36print("# parameter groups in optimiser:", len(opt.param_groups))

Sharing Patterns Across ML

We use hard sharing because (a) it is the simplest, (b) the C-MAPSS backbone is already small, and (c) the gradient-aware methods we propose are most clearly demonstrated on hard sharing. The same techniques apply to soft sharing and mixtures of experts with minor modifications.

The Two Sharing Pitfalls

The point. The shared backbone is the leverage point of MTL. Get its representations right and both tasks benefit; get them wrong and both tasks suffer. The next two sections explain what “getting them right” means: balanced loss combination (4.3) and balanced gradient flow (4.4).

Takeaway

• Shared parameters get gradients from every task. Task-specific parameters only see their own task.
• The full DualTaskModel is ~99% shared. The regression and classification heads add a few thousand parameters on top of a 3.5M backbone.
• PyTorch's named_parameters() reveals the structure. Use it for inspection, freezing, and per-group LR / weight-decay configuration.
• Hard sharing is the default. Soft sharing, mixtures of experts, and adapters are alternatives we touch on but do not adopt.