Angular Dispersion Accelerates $k$-Nearest Neighbors Machine Translation
This work addresses a bottleneck in machine translation for practitioners by improving the efficiency of retrieval in k-NN MT, though it is incremental as it builds on existing approximate lookup methods.
The paper tackles the high computational cost and memory requirements of approximate k-nearest neighbors machine translation (k-NN MT) by proposing to encourage angular dispersion of neural hidden representations of contexts, which accelerates retrieval and slightly improves translations.
Augmenting neural machine translation with external memory at decoding time, in the form of k-nearest neighbors machine translation ($k$-NN MT), is a well-established strategy for increasing translation performance. $k$-NN MT retrieves a set of tokens that occurred in the most similar contexts recorded in a prepared data store, using hidden state representations of translation contexts as vector lookup keys. One of the main disadvantages of this method is the high computational cost and memory requirements. Since an exhaustive search is not feasible in large data stores, practitioners commonly use approximate $k$-NN MT lookup, yet even such algorithms are a bottleneck. In contrast to research directions seeking to accelerate $k$-NN MT by reducing data store size or the number of lookup calls, we pursue an orthogonal direction based on the performance properties of approximate $k$-NN MT lookup data structures. In particular, we propose to encourage angular dispersion of the neural hidden representations of contexts. We show that improving dispersion leads to better balance in the retrieval data structures, accelerating retrieval and slightly improving translations.