AutoScale: Linear Scalarization Guided by Multi-Task Optimization Metrics
This work addresses the challenge of efficient weight selection in multi-task learning for researchers and practitioners, though it is incremental as it builds on existing linear scalarization methods.
The paper tackled the problem of selecting optimal task weights for linear scalarization in multi-task learning by establishing a connection to multi-task optimization metrics, and introduced AutoScale, a framework that uses these metrics to guide weight selection, achieving superior performance with high efficiency across diverse datasets.
Recent multi-task learning studies suggest that linear scalarization, when using well-chosen fixed task weights, can achieve comparable to or even better performance than complex multi-task optimization (MTO) methods. It remains unclear why certain weights yield optimal performance and how to determine these weights without relying on exhaustive hyperparameter search. This paper establishes a direct connection between linear scalarization and MTO methods, revealing through extensive experiments that well-performing scalarization weights exhibit specific trends in key MTO metrics, such as high gradient magnitude similarity. Building on this insight, we introduce AutoScale, a simple yet effective two-phase framework that uses these MTO metrics to guide weight selection for linear scalarization, without expensive weight search. AutoScale consistently shows superior performance with high efficiency across diverse datasets including a new large-scale benchmark.