Simplifying Multi-Task Architectures Through Task-Specific Normalization
This work addresses the problem of architectural complexity in multi-task learning for researchers and practitioners, offering a simpler and more interpretable alternative, though it appears incremental as it builds on existing normalization techniques.
The paper tackles the challenge of balancing resources and mitigating interference in multi-task learning by showing that task-specific normalization layers alone can achieve competitive performance, questioning the need for complex architectures. It introduces Task-Specific Sigmoid Batch Normalization (TSσBN), which matches or exceeds performance on datasets like NYUv2, Cityscapes, CelebA, and PascalContext while being highly parameter-efficient.
Multi-task learning (MTL) aims to leverage shared knowledge across tasks to improve generalization and parameter efficiency, yet balancing resources and mitigating interference remain open challenges. Architectural solutions often introduce elaborate task-specific modules or routing schemes, increasing complexity and overhead. In this work, we show that normalization layers alone are sufficient to address many of these challenges. Simply replacing shared normalization with task-specific variants already yields competitive performance, questioning the need for complex designs. Building on this insight, we propose Task-Specific Sigmoid Batch Normalization (TS$σ$BN), a lightweight mechanism that enables tasks to softly allocate network capacity while fully sharing feature extractors. TS$σ$BN improves stability across CNNs and Transformers, matching or exceeding performance on NYUv2, Cityscapes, CelebA, and PascalContext, while remaining highly parameter-efficient. Moreover, its learned gates provide a natural framework for analyzing MTL dynamics, offering interpretable insights into capacity allocation, filter specialization, and task relationships. Our findings suggest that complex MTL architectures may be unnecessary and that task-specific normalization offers a simple, interpretable, and efficient alternative.