DC-MBR: Distributional Cooling for Minimum Bayesian Risk Decoding
This addresses a specific decoding issue in Neural Machine Translation, offering an incremental improvement for researchers and practitioners in the field.
The paper tackled the poor performance of Minimum Bayesian Risk Decoding (MBR) with label smoothing in Neural Machine Translation, showing it arises from inconsistency between token-level and sequence-level distributions, and proposed Distributional Cooling MBR (DC-MBR) to improve MBR by tuning Softmax temperature, with experiments validating its effectiveness across benchmarks.
Minimum Bayesian Risk Decoding (MBR) emerges as a promising decoding algorithm in Neural Machine Translation. However, MBR performs poorly with label smoothing, which is surprising as label smoothing provides decent improvement with beam search and improves generality in various tasks. In this work, we show that the issue arises from the un-consistency of label smoothing on the token-level and sequence-level distributions. We demonstrate that even though label smoothing only causes a slight change in the token-level, the sequence-level distribution is highly skewed. We coin the issue \emph{autoregressive over-smoothness}. To address this issue, we propose a simple and effective method, Distributional Cooling MBR (DC-MBR), which manipulates the entropy of output distributions by tuning down the Softmax temperature. We theoretically prove the equivalence between pre-tuning label smoothing factor and distributional cooling. Extensive experiments on NMT benchmarks validate that distributional cooling improves MBR in various settings.