Top-$nσ$: Not All Logits Are You Need
This is an incremental improvement for researchers and practitioners in natural language processing, offering a more stable decoding method for reasoning tasks.
The paper tackles the problem of balancing diversity and accuracy in large language model decoding by introducing top-nσ, a sampling method that filters tokens based on a statistical threshold on logits, and it outperforms existing sampling approaches and greedy decoding across four reasoning datasets.
Large language models (LLMs) typically employ greedy decoding or low-temperature sampling for reasoning tasks, reflecting a perceived trade-off between diversity and accuracy. We challenge this convention by introducing top-$nσ$, a novel sampling method that operates directly on pre-softmax logits by leveraging a statistical threshold. Our key insight is that logits naturally separate into a Gaussian-distributed noisy region and a distinct informative region, enabling efficient token filtering without complex probability manipulations. Unlike existing methods (e.g., top-$p$, min-$p$) that inadvertently include more noise tokens at higher temperatures, top-$nσ$ maintains a stable sampling space regardless of temperature scaling. We also provide a theoretical analysis of top-$nσ$ to better understand its behavior. The extensive experimental results across four reasoning-focused datasets demonstrate that our method not only outperforms existing sampling approaches but also surpasses greedy decoding, while maintaining consistent performance even at high temperatures.