Conformal Nucleus Sampling
This work addresses calibration issues in language model decoding for NLP researchers, but it is incremental as it builds on existing methods like conformal prediction.
The authors tackled the problem of whether top-p sampling in language models aligns with its probabilistic meaning across linguistic contexts, finding that OPT models are overconfident and calibration shows moderate inverse scaling with model size.
Language models generate text based on successively sampling the next word. A decoding procedure based on nucleus (top-$p$) sampling chooses from the smallest possible set of words whose cumulative probability exceeds the probability $p$. In this work, we assess whether a top-$p$ set is indeed aligned with its probabilistic meaning in various linguistic contexts. We employ conformal prediction, a calibration procedure that focuses on the construction of minimal prediction sets according to a desired confidence level, to calibrate the parameter $p$ as a function of the entropy of the next word distribution. We find that OPT models are overconfident, and that calibration shows a moderate inverse scaling with model size.