The RoyalFlush System for the WMT 2022 Efficiency Task
This work addresses the efficiency bottleneck in machine translation for real-time applications, representing an incremental improvement through a hybrid approach.
The paper tackled the problem of neural machine translation efficiency by introducing the Hybrid Regression Translation (HRT) system, which improved inference speed by 80% while maintaining equivalent translation performance to autoregressive models, with the fastest system reaching 6k+ words/second on GPU, estimated to be 3.1x faster than the previous year's winner.
This paper describes the submission of the RoyalFlush neural machine translation system for the WMT 2022 translation efficiency task. Unlike the commonly used autoregressive translation system, we adopted a two-stage translation paradigm called Hybrid Regression Translation (HRT) to combine the advantages of autoregressive and non-autoregressive translation. Specifically, HRT first autoregressively generates a discontinuous sequence (e.g., make a prediction every $k$ tokens, $k>1$) and then fills in all previously skipped tokens at once in a non-autoregressive manner. Thus, we can easily trade off the translation quality and speed by adjusting $k$. In addition, by integrating other modeling techniques (e.g., sequence-level knowledge distillation and deep-encoder-shallow-decoder layer allocation strategy) and a mass of engineering efforts, HRT improves 80\% inference speed and achieves equivalent translation performance with the same-capacity AT counterpart. Our fastest system reaches 6k+ words/second on the GPU latency setting, estimated to be about 3.1x faster than the last year's winner.