A Simple Model of Inference Scaling Laws
This work provides a foundational framework for integrating inference scaling with existing scaling laws, which could help optimize prompting costs and performance in reasoning tasks for AI practitioners.
The authors tackled the problem of understanding how performance scales with multiple inference attempts in large language models, proposing a simple statistical model based on memorization that predicts a power law decay in inference loss and aligns with empirical coverage curves in controlled experiments.
Neural scaling laws have garnered significant interest due to their ability to predict model performance as a function of increasing parameters, data, and compute. In this work, we propose a simple statistical ansatz based on memorization to study scaling laws in the context of inference, specifically how performance improves with multiple inference attempts. We explore the coverage, or pass@k metric, which measures the chance of success over repeated attempts and provide a motivation for the observed functional form of the inference scaling behavior of the coverage in large language models (LLMs) on reasoning tasks. We then define an "inference loss", which exhibits a power law decay as the number of trials increases, and connect this result with prompting costs. We further test our construction by conducting experiments on a simple generative model, and find that our predictions are in agreement with the empirical coverage curves in a controlled setting. Our simple framework sets the ground for incorporating inference scaling with other known scaling laws.