DEAS: DEtached value learning with Action Sequence for Scalable Offline RL
This work addresses scalability and performance issues in offline RL for complex, long-horizon tasks, benefiting applications like robotics and vision-language-action models, though it appears incremental as it builds on existing actor-critic and options frameworks.
The paper tackles the challenge of complex, long-horizon sequential decision making in offline reinforcement learning by introducing DEAS, a framework that uses action sequences for value learning and detached value learning to reduce overestimation, resulting in consistent outperformance of baselines on OGBench tasks and significant performance boosts in simulation and real-world manipulation tasks.
Offline reinforcement learning (RL) presents an attractive paradigm for training intelligent agents without expensive online interactions. However, current approaches still struggle with complex, long-horizon sequential decision making. In this work, we introduce DEtached value learning with Action Sequence (DEAS), a simple yet effective offline RL framework that leverages action sequences for value learning. These temporally extended actions provide richer information than single-step actions and can be interpreted through the options framework via semi-Markov decision process Q-learning, enabling reduction of the effective planning horizon by considering longer sequences at once. However, directly adopting such sequences in actor-critic algorithms introduces excessive value overestimation, which we address through detached value learning that steers value estimates toward in-distribution actions that achieve high return in the offline dataset. We demonstrate that DEAS consistently outperforms baselines on complex, long-horizon tasks from OGBench and can be applied to enhance the performance of large-scale Vision-Language-Action models that predict action sequences, significantly boosting performance in both RoboCasa Kitchen simulation tasks and real-world manipulation tasks.