EBBS: An Ensemble with Bi-Level Beam Search for Zero-Shot Machine Translation
This work addresses the challenge of improving translation quality in zero-shot scenarios for multilingual models, representing an incremental advancement over existing ensemble methods.
The paper tackles the problem of noisy and unsatisfactory performance in zero-shot machine translation by proposing EBBS, an ensemble method with a bi-level beam search algorithm, which consistently outperforms direct and pivot translations and existing ensemble techniques on two datasets, and enables distillation to improve inference efficiency without sacrificing quality.
The ability of zero-shot translation emerges when we train a multilingual model with certain translation directions; the model can then directly translate in unseen directions. Alternatively, zero-shot translation can be accomplished by pivoting through a third language (e.g., English). In our work, we observe that both direct and pivot translations are noisy and achieve less satisfactory performance. We propose EBBS, an ensemble method with a novel bi-level beam search algorithm, where each ensemble component explores its own prediction step by step at the lower level but they are synchronized by a "soft voting" mechanism at the upper level. Results on two popular multilingual translation datasets show that EBBS consistently outperforms direct and pivot translations as well as existing ensemble techniques. Further, we can distill the ensemble's knowledge back to the multilingual model to improve inference efficiency; profoundly, our EBBS-based distillation does not sacrifice, or even improves, the translation quality.