Interpretable by Design: Query-Specific Neural Modules for Explainable Reinforcement Learning
This work addresses the need for interpretable and queryable RL systems, with implications for verification and human-AI collaboration, though it is incremental in proposing a new architectural approach.
The paper tackled the problem of making reinforcement learning systems interpretable by designing them to answer diverse queries about their environment, and found that query-specialized architectures achieve near-perfect inference accuracy (99% reachability IoU) even with suboptimal control performance (31% return).
Reinforcement learning has traditionally focused on a singular objective: learning policies that select actions to maximize reward. We challenge this paradigm by asking: what if we explicitly architected RL systems as inference engines that can answer diverse queries about their environment? In deterministic settings, trained agents implicitly encode rich knowledge about reachability, distances, values, and dynamics - yet current architectures are not designed to expose this information efficiently. We introduce Query Conditioned Deterministic Inference Networks (QDIN), a unified architecture that treats different types of queries (policy, reachability, paths, comparisons) as first-class citizens, with specialized neural modules optimized for each inference pattern. Our key empirical finding reveals a fundamental decoupling: inference accuracy can reach near-perfect levels (99% reachability IoU) even when control performance remains suboptimal (31% return), suggesting that the representations needed for accurate world knowledge differ from those required for optimal control. Experiments demonstrate that query specialized architectures outperform both unified models and post-hoc extraction methods, while maintaining competitive control performance. This work establishes a research agenda for RL systems designed from inception as queryable knowledge bases, with implications for interpretability, verification, and human-AI collaboration.