NeoRL: Efficient Exploration for Nonepisodic RL
It addresses efficient exploration in RL without resets, a key challenge for continuous control applications, though it appears incremental as it builds on optimistic principles for nonlinear systems.
The paper tackles nonepisodic reinforcement learning for unknown nonlinear dynamical systems by proposing NeoRL, an optimistic approach using probabilistic models, and achieves a regret bound of O(Γ_T √T) and optimal average cost with minimal regret in experiments.
We study the problem of nonepisodic reinforcement learning (RL) for nonlinear dynamical systems, where the system dynamics are unknown and the RL agent has to learn from a single trajectory, i.e., without resets. We propose Nonepisodic Optimistic RL (NeoRL), an approach based on the principle of optimism in the face of uncertainty. NeoRL uses well-calibrated probabilistic models and plans optimistically w.r.t. the epistemic uncertainty about the unknown dynamics. Under continuity and bounded energy assumptions on the system, we provide a first-of-its-kind regret bound of $O(Γ_T \sqrt{T})$ for general nonlinear systems with Gaussian process dynamics. We compare NeoRL to other baselines on several deep RL environments and empirically demonstrate that NeoRL achieves the optimal average cost while incurring the least regret.