Token-level Ensembling of Models with Different Vocabularies
This addresses a practical bottleneck for researchers and practitioners using open-sourced models with distinct vocabularies, though it is incremental in expanding ensembling techniques.
The paper tackled the problem of ensembling text generation models with different vocabularies, which limits applicability, and proposed an inference-time algorithm that enables token-level ensembling without altering models, resulting in improved translation performance over individual models.
Model ensembling is a technique to combine the predicted distributions of two or more models, often leading to improved robustness and performance. For ensembling in text generation, the next token's probability distribution is derived from a weighted sum of the distributions of each individual model. This requires the underlying models to share the same subword vocabulary, limiting the applicability of ensembling, since many open-sourced models have distinct vocabularies. In research settings, experimentation or upgrades to vocabularies may introduce multiple vocabulary sizes. This paper proposes an inference-time only algorithm that allows for ensembling models with different vocabularies, without the need to learn additional parameters or alter the underlying models. Instead, the algorithm ensures that tokens generated by the ensembled models \textit{agree} in their surface form. We apply this technique to combinations of traditional encoder-decoder models and decoder-only LLMs and evaluate on machine translation. In addition to expanding to model pairs that were previously incapable of token-level ensembling, our algorithm frequently improves translation performance over either model individually.