Analysis of Stochastic Processes through Replay Buffers
This work provides foundational theoretical tools for proving convergence in replay buffer-based algorithms, which are widely used in reinforcement learning.
The paper analyzes the theoretical properties of replay buffers in reinforcement learning by modeling them as systems that transform a stochastic process X into a sampled process Y, examining properties like stationarity and autocorrelation to explain why replay buffers act as de-correlators.
Replay buffers are a key component in many reinforcement learning schemes. Yet, their theoretical properties are not fully understood. In this paper we analyze a system where a stochastic process X is pushed into a replay buffer and then randomly sampled to generate a stochastic process Y from the replay buffer. We provide an analysis of the properties of the sampled process such as stationarity, Markovity and autocorrelation in terms of the properties of the original process. Our theoretical analysis sheds light on why replay buffer may be a good de-correlator. Our analysis provides theoretical tools for proving the convergence of replay buffer based algorithms which are prevalent in reinforcement learning schemes.