Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learning
This addresses the under-studied dependence on training order in multi-domain learning, which is incremental as it builds on existing methods to optimize domain sequencing.
The paper tackles the problem of how training order affects performance in multi-domain learning by analyzing the Lie bracket of gradient vector fields to identify beneficial parameter regions, and validates this on a toy example and bilingual LLM pre-training.
In multi-domain learning, a single model is trained on diverse data domains to leverage shared knowledge and improve generalization. The order in which the data from these domains is used for training can significantly affect the model's performance on each domain. However, this dependence is under-studied. In this paper, we investigate the influence of training order (or data mixing) in multi-domain learning using the concept of Lie bracket of gradient vector fields. By analyzing the infinitesimal effects of changing the training order, we identify regions in the parameter space where altering the order between two training domains can benefit the target loss. We validate the predictions of our theoretical framework on the influence of training order (or data mixing) both on a toy example and bilingual LLM pre-training.