Encoding Sentence Position in Context-Aware Neural Machine Translation with Concatenation
This is an incremental improvement for machine translation researchers, addressing a specific bottleneck in context-aware models.
The paper tackled the problem of improving context-aware neural machine translation by explicitly encoding sentence positions within a concatenated window, finding that certain encoding methods benefited English to Russian translation when trained with a context-discounted loss, but not English to German.
Context-aware translation can be achieved by processing a concatenation of consecutive sentences with the standard Transformer architecture. This paper investigates the intuitive idea of providing the model with explicit information about the position of the sentences contained in the concatenation window. We compare various methods to encode sentence positions into token representations, including novel methods. Our results show that the Transformer benefits from certain sentence position encoding methods on English to Russian translation if trained with a context-discounted loss (Lupo et al., 2022). However, the same benefits are not observed in English to German. Further empirical efforts are necessary to define the conditions under which the proposed approach is beneficial.