Scaling, Simplification, and Adaptation: Lessons from Pretraining on Machine-Translated Text
This addresses the data scarcity issue for lower-resource languages in NLP, offering practical insights for model adaptation, though it is incremental in exploring translation-based pretraining.
The study tackled the problem of limited monolingual data for pretraining in lower-resource languages by using machine-translated text, finding that scaling model capacity improves performance, source-side simplification harms generalization, and adaptation on native text often outperforms native-only models, with specific gains like better accuracy in some tasks but challenges in cultural nuance tasks.
Most languages lack sufficient data for large-scale monolingual pretraining, creating a "data wall." Multilingual pretraining helps but is limited by language imbalance and the "curse of multilinguality." An alternative is to translate high-resource text with machine translation (MT), which raises three questions: (1) How does MT-derived data scale with model capacity? (2) Can source-side transformations (e.g., simplifying English with an LLM) improve generalization to native text? (3) How well do models pretrained on MT-derived data adapt when continually trained on limited native text? We investigate these questions by translating English into Indonesian and Tamil--two typologically distant, lower-resource languages--and pretraining GPT-2 models (124M-774M) on native or MT-derived corpora from raw and LLM-simplified English. We evaluate cross-entropy loss on native text, along with accuracy on syntactic probes and downstream tasks. Our results show that (1) MT-pretrained models benefit from scaling; (2) source-side simplification harms generalization to native text; and (3) adapting MT-pretrained models on native text often yields better performance than native-only models, even with less native data. However, tasks requiring cultural nuance (e.g., toxicity detection) demand more exposure to native data.