VendiRL: A Framework for Self-Supervised Reinforcement Learning of Diversely Diverse Skills
This addresses scalability and evaluation issues in self-supervised RL for agents preparing for unknown tasks, though it appears incremental as it builds on existing diversity measures.
The paper tackled the challenge of learning diverse skills in self-supervised reinforcement learning by introducing VendiRL, a framework that uses the Vendi Score to specify and evaluate any desired form of diversity, resulting in a unified approach for learning diversely diverse skill sets.
In self-supervised reinforcement learning (RL), one of the key challenges is learning a diverse set of skills to prepare agents for unknown future tasks. Despite impressive advances, scalability and evaluation remain prevalent issues. Regarding scalability, the search for meaningful skills can be obscured by high-dimensional feature spaces, where relevant features may vary across downstream task domains. For evaluating skill diversity, defining what constitutes "diversity" typically requires a hard commitment to a specific notion of what it means for skills to be diverse, potentially leading to inconsistencies in how skill diversity is understood, making results across different approaches hard to compare, and leaving many forms of diversity unexplored. To address these issues, we adopt a measure of sample diversity that translates ideas from ecology to machine learning -- the Vendi Score -- allowing the user to specify and evaluate any desired form of diversity. We demonstrate how this metric facilitates skill evaluation and introduce VendiRL, a unified framework for learning diversely diverse sets of skills. Given distinct similarity functions, VendiRL motivates distinct forms of diversity, which could support skill-diversity pretraining in new and richly interactive environments where optimising for various forms of diversity may be desirable.