Semi-Autoregressive Training Improves Mask-Predict Decoding
This work addresses a specific bottleneck in machine translation for researchers and practitioners, offering an incremental improvement to existing methods.
The paper tackled the performance gap between semi-autoregressive and fully autoregressive machine translation models by introducing SMART, a training method that mimics mask-predict decoding, resulting in higher-quality translations that effectively close the gap.
The recently proposed mask-predict decoding algorithm has narrowed the performance gap between semi-autoregressive machine translation models and the traditional left-to-right approach. We introduce a new training method for conditional masked language models, SMART, which mimics the semi-autoregressive behavior of mask-predict, producing training examples that contain model predictions as part of their inputs. Models trained with SMART produce higher-quality translations when using mask-predict decoding, effectively closing the remaining performance gap with fully autoregressive models.