One-stop Training of Multiple Capacity Models
This work addresses the need for efficient model training across different capacity requirements, particularly for multilingual machine translation, though it appears incremental as it builds on existing knowledge distillation concepts.
The paper tackles the problem of training models with varying capacities for different deployment scenarios by proposing a one-stop training framework that jointly trains high- and low-capacity models, achieving comparable or better performance on high-capacity models and outperforming low-capacity baselines on the WMT10 benchmark.
Training models with varying capacities can be advantageous for deploying them in different scenarios. While high-capacity models offer better performance, low-capacity models require fewer computing resources for training and inference. In this work, we propose a novel one-stop training framework to jointly train high-capacity and low-capactiy models. This framework consists of two composite model architectures and a joint training algorithm called Two-Stage Joint-Training (TSJT). Unlike knowledge distillation, where multiple capacity models are trained from scratch separately, our approach integrates supervisions from different capacity models simultaneously, leading to faster and more efficient convergence. Extensive experiments on the multilingual machine translation benchmark WMT10 show that our method outperforms low-capacity baseline models and achieves comparable or better performance on high-capacity models. Notably, the analysis demonstrates that our method significantly influences the initial training process, leading to more efficient convergence and superior solutions.