Towards Reasonably-Sized Character-Level Transformer NMT by Finetuning Subword Systems
This work addresses the problem of inefficient training for character-level NMT for researchers and practitioners, offering a more practical approach, though it is incremental as it builds on existing subword methods.
The authors tackled the challenge of training character-level Transformer NMT models, which typically require deep architectures, by finetuning subword models on characters, resulting in a model that captures morphology better and is more robust to noise, though with slightly worse translation quality.
Applying the Transformer architecture on the character level usually requires very deep architectures that are difficult and slow to train. These problems can be partially overcome by incorporating a segmentation into tokens in the model. We show that by initially training a subword model and then finetuning it on characters, we can obtain a neural machine translation model that works at the character level without requiring token segmentation. We use only the vanilla 6-layer Transformer Base architecture. Our character-level models better capture morphological phenomena and show more robustness to noise at the expense of somewhat worse overall translation quality. Our study is a significant step towards high-performance and easy to train character-based models that are not extremely large.