Dual-Force: Enhanced Offline Diversity Maximization under Imitation Constraints
This work addresses the challenge of offline skill diversity for robotics applications, offering an incremental improvement over prior methods by enhancing skill recall and handling non-stationary rewards.
The paper tackled the problem of offline diversity maximization under imitation constraints by introducing a novel algorithm that uses Van der Waals force and successor features, eliminating the need for a skill discriminator and enabling zero-shot recall of skills. It demonstrated effectiveness in generating diverse skills for robotic tasks like quadruped locomotion and navigation with obstacle traversal, showing stable and efficient training.
While many algorithms for diversity maximization under imitation constraints are online in nature, many applications require offline algorithms without environment interactions. Tackling this problem in the offline setting, however, presents significant challenges that require non-trivial, multi-stage optimization processes with non-stationary rewards. In this work, we present a novel offline algorithm that enhances diversity using an objective based on Van der Waals (VdW) force and successor features, and eliminates the need to learn a previously used skill discriminator. Moreover, by conditioning the value function and policy on a pre-trained Functional Reward Encoding (FRE), our method allows for better handling of non-stationary rewards and provides zero-shot recall of all skills encountered during training, significantly expanding the set of skills learned in prior work. Consequently, our algorithm benefits from receiving a consistently strong diversity signal (VdW), and enjoys more stable and efficient training. We demonstrate the effectiveness of our method in generating diverse skills for two robotic tasks in simulation: locomotion of a quadruped and local navigation with obstacle traversal.