Lite Training Strategies for Portuguese-English and English-Portuguese Translation
This work addresses the problem of expensive model training for machine translation, particularly for Portuguese-English language pairs, though it is incremental as it adapts existing pre-trained models.
The paper tackles the high cost of developing machine translation models by proposing low-cost training strategies for Portuguese-English and English-Portuguese translation, achieving competitive performance with state-of-the-art models using modest hardware like a single 8GB GPU over nine days.
Despite the widespread adoption of deep learning for machine translation, it is still expensive to develop high-quality translation models. In this work, we investigate the use of pre-trained models, such as T5 for Portuguese-English and English-Portuguese translation tasks using low-cost hardware. We explore the use of Portuguese and English pre-trained language models and propose an adaptation of the English tokenizer to represent Portuguese characters, such as diaeresis, acute and grave accents. We compare our models to the Google Translate API and MarianMT on a subset of the ParaCrawl dataset, as well as to the winning submission to the WMT19 Biomedical Translation Shared Task. We also describe our submission to the WMT20 Biomedical Translation Shared Task. Our results show that our models have a competitive performance to state-of-the-art models while being trained on modest hardware (a single 8GB gaming GPU for nine days). Our data, models and code are available at https://github.com/unicamp-dl/Lite-T5-Translation.