Leveraging Human Feedback for Semantically-Relevant Skill Discovery
For reinforcement learning practitioners, this work provides a more feedback-efficient and safer method for skill discovery, though it is incremental over existing preference-based approaches.
The paper addresses the problem of unsafe or misaligned behaviors in unsupervised skill discovery by introducing semantic labelling, a feedback-efficient method that uses human cognitive strengths to label meaningful behaviors. Their proposed approach, SRSD, improves semantic diversity and discovers relevant behaviors across five environments.
Unsupervised skill discovery in reinforcement learning aims to intrinsically motivate agents to discover diverse and useful behaviours. However, unconstrained approaches can produce unsafe, unethical, or misaligned behaviours. To mitigate these risks and improve the practical desireability of discovered skills, recent work grounds the discovery process by leveraging human preference feedback. However, preference-based approaches are feedback-inefficient and inherently ill-equipped to deal with skill spaces composed of a variety of different skills such as running, jumping, walking, etc. To overcome this limitation, we introduce semantic labelling, a novel and feedback-efficient approach that leverages human cognitive strengths to identify and label semantically meaningful behaviours. Based on semantic labelling, we propose Semantically Relevant Skill Discovery (SRSD), a novel human-in-the-loop approach that collects semantic labels from human feedback and learns a reward function to encourage skills to be more semantically diverse and relevant. Through our experiments in a 2D navigation environment and four locomotion environments, we demonstrate that SRSD can improve semantic diversity and discover relevant behaviours while scaling effectively to a large variety of behaviours.