Accelerating Reinforcement Learning Training Using Simulation Surrogate Models
For researchers and practitioners using RL in complex stochastic environments with high-fidelity simulations, this work offers a method to speed up training, though it is incremental as it applies known surrogate modeling ideas to RL.
The paper investigates surrogate models to accelerate reinforcement learning training in dynamic stochastic systems, showing through discrete-event simulation experiments that surrogate models substantially reduce training and retraining time.
High-fidelity simulation models are widely used to analyze complex stochastic systems, but their high computational cost motivates the development of cheaper surrogate models that approximate the simulation model's input-output relationship. In parallel, reinforcement learning (RL) has emerged as a powerful framework for making online decisions in stochastic environments, with increasing attention being given to the use of simulation models as training environments for RL models. We investigate a class of surrogate models suitable for accelerating RL training in settings where the reward structure, model parameters, or system dynamics change over time and explore their interactions with simulation models and RL models. Through numerical experiments on a stochastic service system modeled via discrete-event simulation, we demonstrate that leveraging surrogate models can substantially accelerate RL training and re-training.