Task-Awareness Improves LLM Generations and Uncertainty
This addresses the issue of unreliable LLM predictions in structured tasks for users needing accurate and interpretable outputs, representing a novel method rather than an incremental improvement.
The paper tackles the problem of LLM outputs lacking structural awareness by modeling them in a task-dependent latent space with a dissimilarity measure, enabling Bayes-optimal responses that outperform standard decoding methods like beam search across tasks and improve uncertainty estimation aligned with output quality.
In many applications of LLMs, natural language responses often have an underlying structure such as representing discrete labels, numerical values, or graphs. Yet, existing decoding and uncertainty estimation methods operate only in language space and largely disregard structural information. We address this by modeling LLM outputs directly in a task-dependent latent structure. By equipping this structure with a dissimilarity measure, we can compute Bayes-optimal responses. These are not selected from sampled generations but are newly synthesized by combining individual responses in the latent space. Across different tasks, Bayes-optimal responses consistently outperform standard decoding methods like beam search. Moreover, quantifying uncertainty via the induced Bayesian risk captures variations in terms of the latent structure and improves alignment with output quality and correctness. Our decision-theoretic framework is applicable to any problem that admits a latent response structure and enables reliable task-aware LLM predictions.