OVD-Explorer: Optimism Should Not Be the Sole Pursuit of Exploration in Noisy Environments
This work addresses exploration inefficiency in noisy environments for reinforcement learning practitioners, representing an incremental improvement over existing optimism-based methods.
The paper tackles the problem of inefficient exploration in reinforcement learning due to environmental noise by proposing OVD-Explorer, which reduces over-exploration in high-noise areas and shows superior performance on MuJoCo and GridChaos tasks.
In reinforcement learning, the optimism in the face of uncertainty (OFU) is a mainstream principle for directing exploration towards less explored areas, characterized by higher uncertainty. However, in the presence of environmental stochasticity (noise), purely optimistic exploration may lead to excessive probing of high-noise areas, consequently impeding exploration efficiency. Hence, in exploring noisy environments, while optimism-driven exploration serves as a foundation, prudent attention to alleviating unnecessary over-exploration in high-noise areas becomes beneficial. In this work, we propose Optimistic Value Distribution Explorer (OVD-Explorer) to achieve a noise-aware optimistic exploration for continuous control. OVD-Explorer proposes a new measurement of the policy's exploration ability considering noise in optimistic perspectives, and leverages gradient ascent to drive exploration. Practically, OVD-Explorer can be easily integrated with continuous control RL algorithms. Extensive evaluations on the MuJoCo and GridChaos tasks demonstrate the superiority of OVD-Explorer in achieving noise-aware optimistic exploration.