Ensembling Language Models with Sequential Monte Carlo
This work provides a principled method for combining multiple language models, which is significant for practitioners seeking to improve performance and robustness in various structured text generation tasks by mitigating the sensitivity to individual model and prompt choices.
This paper introduces a unified framework for composing K language models into f-ensemble distributions to address the sensitivity of language model performance to model and prompting choices. They propose a byte-level sequential Monte Carlo algorithm for sampling from these distributions, which allows ensembling models with mismatching vocabularies and achieves consistent sampling. The work demonstrates that alternative aggregation strategies can outperform traditional probability averaging, leading to better ensemble performance.
Practitioners have access to an abundance of language models and prompting strategies for solving many language modeling tasks; yet prior work shows that modeling performance is highly sensitive to both choices. Classical machine learning ensembling techniques offer a principled approach: aggregate predictions from multiple sources to achieve better performance than any single one. However, applying ensembling to language models during decoding is challenging: naively aggregating next-token probabilities yields samples from a locally normalized, biased approximation of the generally intractable ensemble distribution over strings. In this work, we introduce a unified framework for composing $K$ language models into $f$-ensemble distributions for a wide range of functions $f\colon\mathbb{R}_{\geq 0}^{K}\to\mathbb{R}_{\geq 0}$. To sample from these distributions, we propose a byte-level sequential Monte Carlo (SMC) algorithm that operates in a shared character space, enabling ensembles of models with mismatching vocabularies and consistent sampling in the limit. We evaluate a family of $f$-ensembles across prompt and model combinations for various structured text generation tasks, highlighting the benefits of alternative aggregation strategies over traditional probability averaging, and showing that better posterior approximations can yield better ensemble performance.