Option Discovery in the Absence of Rewards with Manifold Analysis
This work addresses the challenge of option discovery for reinforcement learning agents in reward-agnostic settings, offering a novel approach that could enhance exploration and learning efficiency.
The paper tackled the problem of discovering options in reinforcement learning without requiring a reward signal, using spectral graph theory and manifold analysis to develop an algorithm that leverages higher graph frequencies, resulting in clear improvements over competing methods in several domains.
Options have been shown to be an effective tool in reinforcement learning, facilitating improved exploration and learning. In this paper, we present an approach based on spectral graph theory and derive an algorithm that systematically discovers options without access to a specific reward or task assignment. As opposed to the common practice used in previous methods, our algorithm makes full use of the spectrum of the graph Laplacian. Incorporating modes associated with higher graph frequencies unravels domain subtleties, which are shown to be useful for option discovery. Using geometric and manifold-based analysis, we present a theoretical justification for the algorithm. In addition, we showcase its performance in several domains, demonstrating clear improvements compared to competing methods.