What Do Compressed Multilingual Machine Translation Models Forget?
This work addresses the problem of hidden performance drops and bias amplification in compressed models for machine translation, which is important for developers and users in resource-constrained environments, though it is incremental as it builds on existing compression and bias analysis methods.
The study investigated how compression techniques affect multilingual machine translation models, finding that while average BLEU scores slightly decrease, performance drops significantly for under-represented languages and biases in gender and semantics are amplified, with some medium-resource languages improving due to reduced noisy memorization.
Recently, very large pre-trained models achieve state-of-the-art results in various natural language processing (NLP) tasks, but their size makes it more challenging to apply them in resource-constrained environments. Compression techniques allow to drastically reduce the size of the models and therefore their inference time with negligible impact on top-tier metrics. However, the general performance averaged across multiple tasks and/or languages may hide a drastic performance drop on under-represented features, which could result in the amplification of biases encoded by the models. In this work, we assess the impact of compression methods on Multilingual Neural Machine Translation models (MNMT) for various language groups, gender, and semantic biases by extensive analysis of compressed models on different machine translation benchmarks, i.e. FLORES-101, MT-Gender, and DiBiMT. We show that the performance of under-represented languages drops significantly, while the average BLEU metric only slightly decreases. Interestingly, the removal of noisy memorization with compression leads to a significant improvement for some medium-resource languages. Finally, we demonstrate that compression amplifies intrinsic gender and semantic biases, even in high-resource languages. Code: https://github.com/alirezamshi/bias-compressedMT