Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners
This addresses the critical issue of unreliable LLM predictions in robotics, offering a lightweight, scalable solution for safe and autonomous robot planning, though it is incremental in applying conformal prediction to this domain.
The paper tackles the problem of LLM-based planners confidently hallucinating predictions in robotics by introducing KnowNo, a framework that uses conformal prediction to align uncertainty, enabling robots to ask for help when needed. Experiments across simulated and real robot setups show KnowNo improves efficiency and autonomy over baselines while providing formal statistical guarantees on task completion.
Large language models (LLMs) exhibit a wide range of promising capabilities -- from step-by-step planning to commonsense reasoning -- that may provide utility for robots, but remain prone to confidently hallucinated predictions. In this work, we present KnowNo, which is a framework for measuring and aligning the uncertainty of LLM-based planners such that they know when they don't know and ask for help when needed. KnowNo builds on the theory of conformal prediction to provide statistical guarantees on task completion while minimizing human help in complex multi-step planning settings. Experiments across a variety of simulated and real robot setups that involve tasks with different modes of ambiguity (e.g., from spatial to numeric uncertainties, from human preferences to Winograd schemas) show that KnowNo performs favorably over modern baselines (which may involve ensembles or extensive prompt tuning) in terms of improving efficiency and autonomy, while providing formal assurances. KnowNo can be used with LLMs out of the box without model-finetuning, and suggests a promising lightweight approach to modeling uncertainty that can complement and scale with the growing capabilities of foundation models. Website: https://robot-help.github.io