Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control
This work addresses performance instability in continuous control for reinforcement learning practitioners, offering incremental insights into optimization and design.
The paper tackles the instability of deep reinforcement learning agents in continuous control by studying the return landscape, revealing that policy updates can lead to widely varying returns and identifying failure-prone regions. It introduces a distribution-aware procedure that finds stable paths in parameter space, improving policy robustness.
Deep reinforcement learning agents for continuous control are known to exhibit significant instability in their performance over time. In this work, we provide a fresh perspective on these behaviors by studying the return landscape: the mapping between a policy and a return. We find that popular algorithms traverse noisy neighborhoods of this landscape, in which a single update to the policy parameters leads to a wide range of returns. By taking a distributional view of these returns, we map the landscape, characterizing failure-prone regions of policy space and revealing a hidden dimension of policy quality. We show that the landscape exhibits surprising structure by finding simple paths in parameter space which improve the stability of a policy. To conclude, we develop a distribution-aware procedure which finds such paths, navigating away from noisy neighborhoods in order to improve the robustness of a policy. Taken together, our results provide new insight into the optimization, evaluation, and design of agents.