Uniform Loss vs. Specialized Optimization: A Comparative Analysis in Multi-Task Learning
This work addresses the multi-task learning community by clarifying incremental performance claims between optimization methods.
This paper tackles the debate over whether specialized multi-task optimizers (SMTOs) outperform uniform loss weighting in multi-task learning, finding that SMTOs perform well and fixed weights can be competitive, with uniform loss sometimes matching SMTOs.
Specialized Multi-Task Optimizers (SMTOs) balance task learning in Multi-Task Learning by addressing issues like conflicting gradients and differing gradient norms, which hinder equal-weighted task training. However, recent critiques suggest that equally weighted tasks can achieve competitive results compared to SMTOs, arguing that previous SMTO results were influenced by poor hyperparameter optimization and lack of regularization. In this work, we evaluate these claims through an extensive empirical evaluation of SMTOs, including some of the latest methods, on more complex multi-task problems to clarify this behavior. Our findings indicate that SMTOs perform well compared to uniform loss and that fixed weights can achieve competitive performance compared to SMTOs. Furthermore, we demonstrate why uniform loss perform similarly to SMTOs in some instances. The source code is available at https://github.com/Gabriel-SGama/UnitScal_vs_SMTOs.