Multilingual Arbitrage: Optimizing Data Pools to Accelerate Multilingual Progress
This addresses challenges in multilingual NLP by improving performance, especially for low-resource languages, though it is incremental as it builds on existing synthetic data methods.
The paper tackles the problem of model collapse and bias propagation in multilingual synthetic data generation by introducing multilingual arbitrage, which strategically routes samples through a diverse pool of models to capitalize on performance variations, resulting in up to 56.5% improvement in win rates averaged across all languages compared to the best single teacher.
The use of synthetic data has played a critical role in recent state-of-art breakthroughs. However, overly relying on a single oracle teacher model to generate data has been shown to lead to model collapse and invite propagation of biases. These limitations are particularly evident in multilingual settings, where the absence of a universally effective teacher model that excels across all languages presents significant challenges. In this work, we address these extreme difference by introducing "multilingual arbitrage", which capitalizes on performance variations between multiple models for a given language. To do so, we strategically route samples through a diverse pool of models, each with unique strengths in different languages. Across exhaustive experiments on state-of-art models, our work suggests that arbitrage techniques allow for spectacular gains in performance that far outperform relying on a single teacher. In particular, compared to the best single teacher, we observe gains of up to 56.5% improvement in win rates averaged across all languages when switching to multilingual arbitrage. We observe the most significant gains for the least resourced languages in our pool.