Preemptive Detection and Correction of Misaligned Actions in LLM Agents
This addresses a critical safety issue for users deploying LLM agents in real-life applications, such as web shopping, by preventing undesirable consequences, though it is an incremental advancement in agent alignment.
The paper tackles the problem of misalignment between LLM-based agents' behavior and user intent, which can cause negative outcomes like unintended purchases, by introducing InferAct, a method that preemptively detects and corrects misaligned actions before execution, achieving up to 20% improvements in detection accuracy on benchmark tasks.
Deploying LLM-based agents in real-life applications often faces a critical challenge: the misalignment between agents' behavior and user intent. Such misalignment may lead agents to unintentionally execute critical actions that carry negative outcomes (e.g., accidentally triggering a "buy-now" in web shopping), resulting in undesirable or even irreversible consequences. Although addressing these issues is crucial, the preemptive detection and correction of misaligned actions remains relatively underexplored. To fill this gap, we introduce InferAct, a novel approach that leverages the belief reasoning ability of LLMs, grounded in Theory-of-Mind, to detect misaligned actions before execution. Once the misalignment is detected, InferAct alerts users for timely correction, preventing adverse outcomes and enhancing the reliability of LLM agents' decision-making processes. Experiments on three widely used tasks demonstrate that InferAct achieves up to 20% improvements on Marco-F1 against baselines in misaligned action detection. An in-depth evaluation of misalignment correction further highlights InferAct's effectiveness in improving agent alignment.