EMMeTT: Efficient Multimodal Machine Translation Training
This work addresses the problem of training efficiency for multimodal translation models, which is incremental as it builds on existing foundation models like GPT and T5 with a speech encoder.
The paper tackled the challenge of efficiently training multimodal machine translation models by proposing EMMeTT, a framework that improves training efficiency through balanced sampling and novel data handling techniques, resulting in a model that retains text translation capability while outperforming speech translation baselines on four-language subsets of FLORES and FLEURS.
A rising interest in the modality extension of foundation language models warrants discussion on the most effective, and efficient, multimodal training approach. This work focuses on neural machine translation (NMT) and proposes a joint multimodal training regime of Speech-LLM to include automatic speech translation (AST). We investigate two different foundation model architectures, decoder-only GPT and encoder-decoder T5, extended with Canary-1B's speech encoder. To handle joint multimodal training, we propose a novel training framework called EMMeTT. EMMeTT improves training efficiency with the following: balanced sampling across languages, datasets, and modalities; efficient sequential data iteration; and a novel 2D bucketing scheme for multimodal data, complemented by a batch size optimizer (OOMptimizer). We show that a multimodal training consistently helps with both architectures. Moreover, SALM-T5 trained with EMMeTT retains the original NMT capability while outperforming AST baselines on four-language subsets of FLORES and FLEURS. The resultant Multimodal Translation Model produces strong text and speech translation results at the same time.