Can Optimization Trajectories Explain Multi-Task Transfer?
This work addresses a fundamental issue in deep learning for researchers and practitioners using multi-task training, but it is incremental as it builds on prior conjectures without proposing a new solution.
The paper tackled the problem of understanding why multi-task learning (MTL) often fails to improve generalization, by empirically studying how MTL impacts optimization trajectories and their correlation with generalization gaps. The results showed that MTL leads to early generalization gaps compared to single-task training, but factors like gradient conflict do not predict generalization, raising questions about existing optimization methods.
Despite the widespread adoption of multi-task training in deep learning, little is understood about how multi-task learning (MTL) affects generalization. Prior work has conjectured that the negative effects of MTL are due to optimization challenges that arise during training, and many optimization methods have been proposed to improve multi-task performance. However, recent work has shown that these methods fail to consistently improve multi-task generalization. In this work, we seek to improve our understanding of these failures by empirically studying how MTL impacts the optimization of tasks, and whether this impact can explain the effects of MTL on generalization. We show that MTL results in a generalization gap (a gap in generalization at comparable training loss) between single-task and multi-task trajectories early into training. However, we find that factors of the optimization trajectory previously proposed to explain generalization gaps in single-task settings cannot explain the generalization gaps between single-task and multi-task models. Moreover, we show that the amount of gradient conflict between tasks is correlated with negative effects to task optimization, but is not predictive of generalization. Our work sheds light on the underlying causes for failures in MTL and, importantly, raises questions about the role of general purpose multi-task optimization algorithms.