Efficiently Reusing Old Models Across Languages via Transfer Learning
This work addresses the financial and environmental costs of training NMT models, particularly in transfer learning, by enabling efficient reuse of old models across languages, though it is incremental as it builds on existing transfer learning approaches.
The paper tackles the high cost of training neural machine translation models by proposing a method to reuse an existing trained model for different language pairs without architectural changes, achieving better translation quality and shorter convergence times compared to training from scratch.
Recent progress in neural machine translation is directed towards larger neural networks trained on an increasing amount of hardware resources. As a result, NMT models are costly to train, both financially, due to the electricity and hardware cost, and environmentally, due to the carbon footprint. It is especially true in transfer learning for its additional cost of training the "parent" model before transferring knowledge and training the desired "child" model. In this paper, we propose a simple method of re-using an already trained model for different language pairs where there is no need for modifications in model architecture. Our approach does not need a separate parent model for each investigated language pair, as it is typical in NMT transfer learning. To show the applicability of our method, we recycle a Transformer model trained by different researchers and use it to seed models for different language pairs. We achieve better translation quality and shorter convergence times than when training from random initialization.