Opinion: Towards Unified Expressive Policy Optimization for Robust Robot Learning
This work addresses robust robot learning for safer and more efficient policy deployment, representing a novel method rather than an incremental improvement.
The paper tackles the challenges of limited multimodal behavior coverage and distributional shifts in offline-to-online reinforcement learning for robotics by proposing UEPO, a unified generative framework, achieving +5.9% improvement on locomotion tasks and +12.4% on dexterous manipulation over Uni-O4 on the D4RL benchmark.
Offline-to-online reinforcement learning (O2O-RL) has emerged as a promising paradigm for safe and efficient robotic policy deployment but suffers from two fundamental challenges: limited coverage of multimodal behaviors and distributional shifts during online adaptation. We propose UEPO, a unified generative framework inspired by large language model pretraining and fine-tuning strategies. Our contributions are threefold: (1) a multi-seed dynamics-aware diffusion policy that efficiently captures diverse modalities without training multiple models; (2) a dynamic divergence regularization mechanism that enforces physically meaningful policy diversity; and (3) a diffusion-based data augmentation module that enhances dynamics model generalization. On the D4RL benchmark, UEPO achieves +5.9\% absolute improvement over Uni-O4 on locomotion tasks and +12.4\% on dexterous manipulation, demonstrating strong generalization and scalability.