STACHE: Local Black-Box Explanations for Reinforcement Learning Policies
This addresses the need for reliable debugging and verification tools in sparse-reward or safety-critical environments, offering actionable insights for developers and researchers.
The paper tackled the problem of unexpected behavior in reinforcement learning agents by proposing STACHE, a framework for generating local black-box explanations for actions in discrete Markov games, which produced robustness regions and minimal counterfactuals to provide insights into agent sensitivity and decision boundaries during training.
Reinforcement learning agents often behave unexpectedly in sparse-reward or safety-critical environments, creating a strong need for reliable debugging and verification tools. In this paper, we propose STACHE, a comprehensive framework for generating local, black-box explanations for an agent's specific action within discrete Markov games. Our method produces a Composite Explanation consisting of two complementary components: (1) a Robustness Region, the connected neighborhood of states where the agent's action remains invariant, and (2) Minimal Counterfactuals, the smallest state perturbations required to alter that decision. By exploiting the structure of factored state spaces, we introduce an exact, search-based algorithm that circumvents the fidelity gaps of surrogate models. Empirical validation on Gymnasium environments demonstrates that our framework not only explains policy actions, but also effectively captures the evolution of policy logic during training - from erratic, unstable behavior to optimized, robust strategies - providing actionable insights into agent sensitivity and decision boundaries.