Behavioral Mode Discovery for Fine-tuning Multimodal Generative Policies
For roboticists fine-tuning generative policies, this method addresses the problem of behavioral diversity collapse, offering a way to maintain multimodality while improving task performance.
The paper proposes an unsupervised mode discovery framework to fine-tune multimodal generative policies with RL, preventing behavioral collapse while improving task success. On robotic manipulation tasks, the method achieves higher success rates and preserves richer multimodal action distributions compared to conventional fine-tuning.
We address the problem of fine-tuning pre-trained generative policies with reinforcement learning (RL) while preserving the multimodality of their action distributions. Existing methods for RL fine-tuning of generative policies (e.g., diffusion policies) improve task performance but often collapse diverse behaviors into a single reward-maximizing mode. To mitigate this issue, we propose an unsupervised mode discovery framework that uncovers latent behavioral modes within generative policies. The discovered modes enable the use of mutual information as an intrinsic reward, regularizing RL fine-tuning to enhance task success while maintaining behavioral diversity. Experiments on robotic manipulation tasks demonstrate that our method consistently outperforms conventional fine-tuning approaches, achieving higher success rates and preserving richer multimodal action distributions.