RAD: Retrieval High-quality Demonstrations to Enhance Decision-making
This addresses a key bottleneck in offline RL for improving decision-making in data-scarce environments, though it appears incremental as it builds on existing retrieval and generative modeling techniques.
The paper tackles the challenge of dataset sparsity and lack of transition overlap in offline reinforcement learning, which hinders long-horizon planning, by proposing RAD, a method that retrieves high-quality demonstrations and uses diffusion models for planning, achieving competitive or superior performance across diverse benchmarks.
Offline reinforcement learning (RL) enables agents to learn policies from fixed datasets, avoiding costly or unsafe environment interactions. However, its effectiveness is often limited by dataset sparsity and the lack of transition overlap between suboptimal and expert trajectories, which makes long-horizon planning particularly challenging. Prior solutions based on synthetic data augmentation or trajectory stitching often fail to generalize to novel states and rely on heuristic stitching points. To address these challenges, we propose Retrieval High-quAlity Demonstrations (RAD) for decision-making, which combines non-parametric retrieval with diffusion-based generative modeling. RAD dynamically retrieves high-return states from the offline dataset as target states based on state similarity and return estimation, and plans toward them using a condition-guided diffusion model. Such retrieval-guided generation enables flexible trajectory stitching and improves generalization when encountered with underrepresented or out-of-distribution states. Extensive experiments confirm that RAD achieves competitive or superior performance compared to baselines across diverse benchmarks, validating its effectiveness.