Intrinsic Credit Assignment for Long Horizon Interaction
This addresses the challenge of long-horizon uncertainty in reinforcement learning for applications like customer service and personalization, representing a novel method for a known bottleneck.
The paper tackles the problem of training agents to navigate uncertainty over long horizons by proposing ΔBelief-RL, which uses changes in a language model's intrinsic beliefs to reward intermediate progress for credit assignment. The method outperforms outcome-based rewards in reinforcement learning, with improvements generalizing to out-of-distribution applications and performance scaling beyond training horizons.
How can we train agents to navigate uncertainty over long horizons? In this work, we propose ΔBelief-RL, which leverages a language model's own intrinsic beliefs to reward intermediate progress. Our method utilizes the change in the probability an agent assigns to the target solution for credit assignment. By training on synthetic interaction data, ΔBelief-RL teaches information-seeking capabilities that consistently outperform purely outcome-based rewards for Reinforcement Learning, with improvements generalizing to out-of-distribution applications ranging from customer service to personalization. Notably, the performance continues to improve as we scale test-time interactions beyond the training horizon, with interaction-efficiency increasing even on Pass@k metrics. Overall, our work introduces a scalable training strategy for navigating uncertainty over a long-horizon, by enabling credit assignment to intermediate actions via intrinsic ΔBelief rewards.