Guided Speculative Inference for Efficient Test-Time Alignment of LLMs
This work addresses the challenge of improving test-time alignment for LLMs efficiently, offering a novel algorithmic approach with potential applications in domains requiring high reasoning accuracy, though it appears incremental in advancing speculative decoding techniques.
The paper tackles the problem of efficient reward-guided decoding in large language models by proposing Guided Speculative Inference (GSI), which combines soft best-of-n scaling with a reward model and speculative samples from an auxiliary model, achieving higher accuracy than existing methods on reasoning benchmarks like MATH500 and GSM8K.
We propose Guided Speculative Inference (GSI), a novel algorithm for efficient reward-guided decoding in large language models. GSI combines soft best-of-$n$ test-time scaling with a reward model $r(x,y)$ and speculative samples from a small auxiliary model $π_S(y\mid x)$. We provably approximate both the optimal tilted policy $π_{β,B}(y\mid x) \propto π_B(y\mid x)\exp(β\,r(x,y))$ of soft best-of-$n$ under the base model $π_B$, as well as the expected reward under the optimal policy. In experiments on reasoning benchmarks (MATH500, OlympiadBench, Minerva Math, MMLU-STEM, GSM8K), our method achieves higher accuracy than standard soft best-of-$n$ with $π_S$ and reward-guided speculative decoding (Liao et al., 2025), and in certain settings even outperforms soft best-of-$n$ with $π_B$. The code is available at https://github.com/j-geuter/GSI .