Scalarization for Multi-Task and Multi-Domain Learning at Scale
This work addresses the challenge of optimizing multi-task and multi-domain learning, which is important for improving model efficiency and knowledge transfer in machine learning applications, though it is incremental in nature.
The paper investigates the training dynamics of multi-task and multi-domain networks, finding that uniform scalarization performs comparably to state-of-the-art methods, and proposes using population-based training to optimize scalarization weights for large-scale scenarios.
Training a single model on multiple input domains and/or output tasks allows for compressing information from multiple sources into a unified backbone hence improves model efficiency. It also enables potential positive knowledge transfer across tasks/domains, leading to improved accuracy and data-efficient training. However, optimizing such networks is a challenge, in particular due to discrepancies between the different tasks or domains: Despite several hypotheses and solutions proposed over the years, recent work has shown that uniform scalarization training, i.e., simply minimizing the average of the task losses, yields on-par performance with more costly SotA optimization methods. This raises the issue of how well we understand the training dynamics of multi-task and multi-domain networks. In this work, we first devise a large-scale unified analysis of multi-domain and multi-task learning to better understand the dynamics of scalarization across varied task/domain combinations and model sizes. Following these insights, we then propose to leverage population-based training to efficiently search for the optimal scalarization weights when dealing with a large number of tasks or domains.