Focused Skill Discovery: Learning to Control Specific State Variables while Minimizing Side Effects
This work addresses a domain-specific issue in reinforcement learning by improving skill discovery for more efficient exploration and reduced side effects, representing an incremental advancement over existing methods.
The paper tackles the problem of skill discovery algorithms lacking control over specific state variables, which hampers exploration and causes side effects in downstream tasks, by introducing a method that learns focused skills to control these variables, resulting in a threefold improvement in state space coverage and automatic avoidance of negative side effects.
Skills are essential for unlocking higher levels of problem solving. A common approach to discovering these skills is to learn ones that reliably reach different states, thus empowering the agent to control its environment. However, existing skill discovery algorithms often overlook the natural state variables present in many reinforcement learning problems, meaning that the discovered skills lack control of specific state variables. This can significantly hamper exploration efficiency, make skills more challenging to learn with, and lead to negative side effects in downstream tasks when the goal is under-specified. We introduce a general method that enables these skill discovery algorithms to learn focused skills -- skills that target and control specific state variables. Our approach improves state space coverage by a factor of three, unlocks new learning capabilities, and automatically avoids negative side effects in downstream tasks.