Talk Less, Verify More: Improving LLM Assistants with Semantic Checks and Execution Feedback
This work addresses the need for more reliable GenAI systems in enterprise decision support, though it appears incremental by building on existing generator-discriminator frameworks.
The paper tackled the problem of unreliable outputs from LLM assistants in enterprise workflows by introducing verification techniques Q* and Feedback+, which reduced error rates and task completion time on benchmark datasets like Spider, Bird, and GSM8K.
As large language model (LLM) assistants become increasingly integrated into enterprise workflows, their ability to generate accurate, semantically aligned, and executable outputs is critical. However, current conversational business analytics (CBA) systems often lack built-in verification mechanisms, leaving users to manually validate potentially flawed results. This paper introduces two complementary verification techniques: Q*, which performs reverse translation and semantic matching between code and user intent, and Feedback+, which incorporates execution feedback to guide code refinement. Embedded within a generator-discriminator framework, these mechanisms shift validation responsibilities from users to the system. Evaluations on three benchmark datasets, Spider, Bird, and GSM8K, demonstrate that both Q* and Feedback+ reduce error rates and task completion time. The study also identifies reverse translation as a key bottleneck, highlighting opportunities for future improvement. Overall, this work contributes a design-oriented framework for building more reliable, enterprise-grade GenAI systems capable of trustworthy decision support.