Connecting and Comparing Language Model Interpolation Techniques
This work addresses the problem of improving language model performance for researchers and practitioners, but it is incremental as it builds on prior comparisons and focuses on specific techniques.
The paper connects and compares language model interpolation techniques, showing that count merging and Bayesian interpolation outperform linear interpolation and perform similarly to each other in scenarios with abundant training data.
In this work, we uncover a theoretical connection between two language model interpolation techniques, count merging and Bayesian interpolation. We compare these techniques as well as linear interpolation in three scenarios with abundant training data per component model. Consistent with prior work, we show that both count merging and Bayesian interpolation outperform linear interpolation. We include the first (to our knowledge) published comparison of count merging and Bayesian interpolation, showing that the two techniques perform similarly. Finally, we argue that other considerations will make Bayesian interpolation the preferred approach in most circumstances.