Taming OOD Actions for Offline Reinforcement Learning: An Advantage-Based Approach
This addresses a key limitation in offline RL for improving policy generalization in real-world applications where online interaction is costly, though it is an incremental advance over existing conservative methods.
The paper tackles the problem of distribution shift in offline reinforcement learning by proposing an advantage-based approach to evaluate out-of-distribution actions discriminatively, achieving state-of-the-art performance on the D4RL benchmark with strong gains on challenging tasks.
Offline reinforcement learning (RL) learns policies from fixed datasets without online interactions, but suffers from distribution shift, causing inaccurate evaluation and overestimation of out-of-distribution (OOD) actions. Existing methods counter this by conservatively discouraging all OOD actions, which limits generalization. We propose Advantage-based Diffusion Actor-Critic (ADAC), which evaluates OOD actions via an advantage-like function and uses it to modulate the Q-function update discriminatively. Our key insight is that the (state) value function is generally learned more reliably than the action-value function; we thus use the next-state value to indirectly assess each action. We develop a PointMaze environment to clearly visualize that advantage modulation effectively selects superior OOD actions while discouraging inferior ones. Moreover, extensive experiments on the D4RL benchmark show that ADAC achieves state-of-the-art performance, with especially strong gains on challenging tasks.