Optimizing Temperature for Language Models with Multi-Sample Inference
This work addresses a key challenge in optimizing the performance of large language models, which is crucial for natural language processing tasks and applications.
The authors tackled the problem of temperature selection in large language models, achieving improved performance by proposing a novel entropy-based metric for automated temperature optimization, which consistently outperforms fixed-temperature baselines. The result is a comprehensive analysis of temperature's role in performance optimization across various models and tasks.
Multi-sample aggregation strategies, such as majority voting and best-of-N sampling, are widely used in contemporary large language models (LLMs) to enhance predictive accuracy across various tasks. A key challenge in this process is temperature selection, which significantly impacts model performance. Existing approaches either rely on a fixed default temperature or require labeled validation data for tuning, which are often scarce and difficult to obtain. This paper addresses the challenge of automatically identifying the (near)-optimal temperature for different LLMs using multi-sample aggregation strategies, without relying on task-specific validation data. We provide a comprehensive analysis of temperature's role in performance optimization, considering variations in model architectures, datasets, task types, model sizes, and predictive accuracy. Furthermore, we propose a novel entropy-based metric for automated temperature optimization, which consistently outperforms fixed-temperature baselines. Additionally, we incorporate a stochastic process model to enhance interpretability, offering deeper insights into the relationship between temperature and model performance.