Building surrogate models using trajectories of agents trained by Reinforcement Learning
This work addresses the challenge of building accurate surrogate models for complex simulators, which is crucial for enabling efficient reinforcement learning policy optimization in domains like engineering or robotics, though it is incremental as it builds on existing sampling strategies.
The paper tackles the problem of sample efficiency in surrogate modeling for computationally expensive simulations with wide state spaces by proposing a method that uses trajectories from reinforcement learning-trained agents to sample deterministic environments. The result shows that a mixed dataset combining random, expert, and entropy-maximizing agents achieves the best scores across all datasets, improving state-of-the-art performance.
Sample efficiency in the face of computationally expensive simulations is a common concern in surrogate modeling. Current strategies to minimize the number of samples needed are not as effective in simulated environments with wide state spaces. As a response to this challenge, we propose a novel method to efficiently sample simulated deterministic environments by using policies trained by Reinforcement Learning. We provide an extensive analysis of these surrogate-building strategies with respect to Latin-Hypercube sampling or Active Learning and Kriging, cross-validating performances with all sampled datasets. The analysis shows that a mixed dataset that includes samples acquired by random agents, expert agents, and agents trained to explore the regions of maximum entropy of the state transition distribution provides the best scores through all datasets, which is crucial for a meaningful state space representation. We conclude that the proposed method improves the state-of-the-art and clears the path to enable the application of surrogate-aided Reinforcement Learning policy optimization strategies on complex simulators.