Stochastic Resetting Accelerates Policy Convergence in Reinforcement Learning
This addresses the challenge of inefficient exploration and slow learning in reinforcement learning, particularly for sparse-reward environments, by adapting a statistical mechanics concept, though it builds incrementally on existing resetting theory.
The paper tackles the problem of slow policy convergence in reinforcement learning by introducing stochastic resetting, which intermittently returns the agent to a fixed state. The result shows accelerated convergence in both tabular grid environments and continuous control tasks, with improvements in deep reinforcement learning for sparse-reward settings.
Stochastic resetting, where a dynamical process is intermittently returned to a fixed reference state, has emerged as a powerful mechanism for optimizing first-passage properties. Existing theory largely treats static, non-learning processes. Here we ask how stochastic resetting interacts with reinforcement learning, where the underlying dynamics adapt through experience. In tabular grid environments, we find that resetting accelerates policy convergence even when it does not reduce the search time of a purely diffusive agent, indicating a novel mechanism beyond classical first-passage optimization. In a continuous control task with neural-network-based value approximation, we show that random resetting improves deep reinforcement learning when exploration is difficult and rewards are sparse. Unlike temporal discounting, resetting preserves the optimal policy while accelerating convergence by truncating long, uninformative trajectories to enhance value propagation. Our results establish stochastic resetting as a simple, tunable mechanism for accelerating learning, translating a canonical phenomenon of statistical mechanics into an optimization principle for reinforcement learning.