Steady State Analysis of Episodic Reinforcement Learning
This work addresses the theoretical gap in episodic RL by establishing steady-state properties, offering a unified framework for RL formalisms and practical data collection improvements.
The paper proves that episodic reinforcement learning environments have a unique steady state under any behavior policy, with the agent's input distribution converging to it, and applies this insight to unify episodic and continual RL, demonstrating practical benefits like a steady-state policy gradient theorem and a perturbation method for faster convergence.
This paper proves that the episodic learning environment of every finite-horizon decision task has a unique steady state under any behavior policy, and that the marginal distribution of the agent's input indeed converges to the steady-state distribution in essentially all episodic learning processes. This observation supports an interestingly reversed mindset against conventional wisdom: While the existence of unique steady states was often presumed in continual learning but considered less relevant in episodic learning, it turns out their existence is guaranteed for the latter. Based on this insight, the paper unifies episodic and continual RL around several important concepts that have been separately treated in these two RL formalisms. Practically, the existence of unique and approachable steady state enables a general way to collect data in episodic RL tasks, which the paper applies to policy gradient algorithms as a demonstration, based on a new steady-state policy gradient theorem. Finally, the paper also proposes and experimentally validates a perturbation method that facilitates rapid steady-state convergence in real-world RL tasks.