Beyond Reward: A Bounded Measure of Agent Environment Coupling
This addresses the challenge of reliable deployment for RL systems by providing a novel monitoring tool for early detection of interaction degradation, though it is incremental in improving existing monitoring approaches.
The paper tackled the problem of detecting early failures in reinforcement learning agents under distribution shifts by introducing bipredictability as a real-time measure of agent-environment coupling, showing that it detects 89.3% of perturbations with 4.4x lower latency compared to reward-based methods.
Real-world reinforcement learning (RL) agents operate in closed-loop systems where actions shape future observations, making reliable deployment under distribution shifts a persistent challenge. Existing monitoring relies on reward or task metrics, capturing outcomes but missing early coupling failures. We introduce bipredictability (P) as the ratio of shared information in the observation, action, outcome loop to the total available information, a principled, real time measure of interaction effectiveness with provable bounds, comparable across tasks. An auxiliary monitor, the Information Digital Twin (IDT), computes P and its diagnostic components from the interaction stream. We evaluate SAC and PPO agents on MuJoCo HalfCheetah under eight agent, and environment-side perturbations across 168 trials. Under nominal operation, agents exhibit P = 0.33 plus minus 0.02, below the classical bound of 0.5, revealing an informational cost of action selection. The IDT detects 89.3% of perturbations versus 44.0% for reward based monitoring, with 4.4x lower median latency. Bipredictability enables early detection of interaction degradation before performance drops and provides a prerequisite signal for closed loop self regulation in deployed RL systems.