CA-BED: Conversation-Aware Bayesian Experimental Design
For developers of conversational AI systems, CA-BED provides a principled method to enhance information-seeking dialogues, though it is an incremental extension of existing Bayesian methods to LLMs.
CA-BED improves LLM performance in interactive question-asking tasks by integrating Bayesian Experimental Design with LLM-based likelihood estimation, achieving a 21.8% average increase in success rates over direct prompting with only 1.8 additional turns.
Large Language Models (LLMs) excel at static reasoning tasks, yet their performance often degrades in interactive scenarios where information must be actively acquired through questioning. A key challenge lies in selecting questions that reduce uncertainty while incorporating responses that may be ambiguous or only partially informative. To address this, we propose Conversation-Aware Bayesian Experimental Design (CA-BED), an inference-time probabilistic dialog planning framework that integrates Bayesian Experimental Design with LLM-based likelihood estimation to optimize question selection over multiple conversational turns. CA-BED maintains a belief distribution over hypotheses, anticipates possible answers, and propagates expected information gain through a simulated conversation tree. Across two structured entity-deduction benchmarks, CA-BED yields an average 21.8% improvement in success rates over direct prompting, with comparable gains relative to alternative information-seeking methods. It achieves these gains with an average increase of only 1.8 conversational turns compared to direct prompting.