Geometric Active Exploration in Markov Decision Processes: the Benefit of Abstraction
This work addresses the problem of inefficient experiment design in scientific discovery for researchers using reinforcement learning, offering a novel approach that is incremental in bridging existing frameworks.
The paper tackles the scalability issue in Active Exploration for designing experiments in dynamical systems by leveraging geometric structures like permutation invariance, proposing the Geometric Active Exploration algorithm that improves sample efficiency with theoretical and experimental validation.
How can a scientist use a Reinforcement Learning (RL) algorithm to design experiments over a dynamical system's state space? In the case of finite and Markovian systems, an area called Active Exploration (AE) relaxes the optimization problem of experiments design into Convex RL, a generalization of RL admitting a wider notion of reward. Unfortunately, this framework is currently not scalable and the potential of AE is hindered by the vastness of experiment spaces typical of scientific discovery applications. However, these spaces are often endowed with natural geometries, e.g., permutation invariance in molecular design, that an agent could leverage to improve the statistical and computational efficiency of AE. To achieve this, we bridge AE and MDP homomorphisms, which offer a way to exploit known geometric structures via abstraction. Towards this goal, we make two fundamental contributions: we extend MDP homomorphisms formalism to Convex RL, and we present, to the best of our knowledge, the first analysis that formally captures the benefit of abstraction via homomorphisms on sample efficiency. Ultimately, we propose the Geometric Active Exploration (GAE) algorithm, which we analyse theoretically and experimentally in environments motivated by problems in scientific discovery.