Uncertainty Quantification and Decomposition for LLM-based Recommendation
This work addresses the need for trustworthy LLM-based recommendations, though it is incremental as it builds on existing uncertainty quantification methods applied to a specific domain.
The paper tackles the problem of uncertainty in LLM-based recommendations by introducing a framework to estimate and decompose predictive uncertainty, showing it effectively indicates reliability and proposing uncertainty-aware prompting to reduce uncertainty and enhance recommendations.
Despite the widespread adoption of large language models (LLMs) for recommendation, we demonstrate that LLMs often exhibit uncertainty in their recommendations. To ensure the trustworthy use of LLMs in generating recommendations, we emphasize the importance of assessing the reliability of recommendations generated by LLMs. We start by introducing a novel framework for estimating the predictive uncertainty to quantitatively measure the reliability of LLM-based recommendations. We further propose to decompose the predictive uncertainty into recommendation uncertainty and prompt uncertainty, enabling in-depth analyses of the primary source of uncertainty. Through extensive experiments, we (1) demonstrate predictive uncertainty effectively indicates the reliability of LLM-based recommendations, (2) investigate the origins of uncertainty with decomposed uncertainty measures, and (3) propose uncertainty-aware prompting for a lower predictive uncertainty and enhanced recommendation. Our source code and model weights are available at https://github.com/WonbinKweon/UNC_LLM_REC_WWW2025