AutoQD: Automatic Discovery of Diverse Behaviors with Quality-Diversity Optimization
This addresses the need for automated behavior discovery in reinforcement learning and QD optimization, reducing reliance on domain-specific knowledge, though it builds incrementally on existing QD methods.
The paper tackles the problem of Quality-Diversity (QD) algorithms relying on hand-crafted behavioral descriptors by introducing AutoQD, which automatically generates descriptors using embeddings of policy occupancy measures, enabling discovery of diverse policies without predefined notions of diversity in continuous control tasks.
Quality-Diversity (QD) algorithms have shown remarkable success in discovering diverse, high-performing solutions, but rely heavily on hand-crafted behavioral descriptors that constrain exploration to predefined notions of diversity. Leveraging the equivalence between policies and occupancy measures, we present a theoretically grounded approach to automatically generate behavioral descriptors by embedding the occupancy measures of policies in Markov Decision Processes. Our method, AutoQD, leverages random Fourier features to approximate the Maximum Mean Discrepancy (MMD) between policy occupancy measures, creating embeddings whose distances reflect meaningful behavioral differences. A low-dimensional projection of these embeddings that captures the most behaviorally significant dimensions is then used as behavioral descriptors for off-the-shelf QD methods. We prove that our embeddings converge to true MMD distances between occupancy measures as the number of sampled trajectories and embedding dimensions increase. Through experiments in multiple continuous control tasks we demonstrate AutoQD's ability in discovering diverse policies without predefined behavioral descriptors, presenting a well-motivated alternative to prior methods in unsupervised Reinforcement Learning and QD optimization. Our approach opens new possibilities for open-ended learning and automated behavior discovery in sequential decision making settings without requiring domain-specific knowledge.