Chunk-Guided Q-Learning
This addresses a key bottleneck in offline RL for improving long-horizon decision-making, though it appears incremental as it builds on existing action-chunked methods.
The paper tackles the problem of bootstrapping error accumulation in offline reinforcement learning by proposing Chunk-Guided Q-Learning, which combines single-step and chunk-based critics to reduce error while preserving fine-grained value propagation, achieving strong performance on long-horizon tasks.
In offline reinforcement learning (RL), single-step temporal-difference (TD) learning can suffer from bootstrapping error accumulation over long horizons. Action-chunked TD methods mitigate this by backing up over multiple steps, but can introduce suboptimality by restricting the policy class to open-loop action sequences. To resolve this trade-off, we present Chunk-Guided Q-Learning (CGQ), a single-step TD algorithm that guides a fine-grained single-step critic by regularizing it toward a chunk-based critic trained using temporally extended backups. This reduces compounding error while preserving fine-grained value propagation. We theoretically show that CGQ attains tighter critic optimality bounds than either single-step or action-chunked TD learning alone. Empirically, CGQ achieves strong performance on challenging long-horizon OGBench tasks, often outperforming both single-step and action-chunked methods.