Learning to Seek Evidence: A Verifiable Reasoning Agent with Causal Faithfulness Analysis
This addresses the need for trustworthy AI in domains like medicine by providing a framework for verifiable reasoning, though it is incremental in improving existing explanation methods.
The paper tackles the problem of unverifiable AI explanations in high-stakes domains by proposing an interactive agent that learns to seek visual evidence for diagnostic reasoning, resulting in an 18% reduction in Brier score compared to a baseline and demonstrating faithfulness through causal interventions.
Explanations for AI models in high-stakes domains like medicine often lack verifiability, which can hinder trust. To address this, we propose an interactive agent that produces explanations through an auditable sequence of actions. The agent learns a policy to strategically seek external visual evidence to support its diagnostic reasoning. This policy is optimized using reinforcement learning, resulting in a model that is both efficient and generalizable. Our experiments show that this action-based reasoning process significantly improves calibrated accuracy, reducing the Brier score by 18\% compared to a non-interactive baseline. To validate the faithfulness of the agent's explanations, we introduce a causal intervention method. By masking the visual evidence the agent chooses to use, we observe a measurable degradation in its performance ($Δ$Brier=+0.029), confirming that the evidence is integral to its decision-making process. Our work provides a practical framework for building AI systems with verifiable and faithful reasoning capabilities.