Fractal Landscapes in Policy Optimization
This addresses a fundamental limitation in RL training for continuous domains, though it is incremental as it builds on existing chaos theory techniques.
The paper tackles the problem of policy gradient failures in deep reinforcement learning by showing that the optimization landscape can be fractal, lacking gradients in certain MDPs, and it develops a method to estimate local smoothness to identify these cases.
Policy gradient lies at the core of deep reinforcement learning (RL) in continuous domains. Despite much success, it is often observed in practice that RL training with policy gradient can fail for many reasons, even on standard control problems with known solutions. We propose a framework for understanding one inherent limitation of the policy gradient approach: the optimization landscape in the policy space can be extremely non-smooth or fractal for certain classes of MDPs, such that there does not exist gradient to be estimated in the first place. We draw on techniques from chaos theory and non-smooth analysis, and analyze the maximal Lyapunov exponents and Hölder exponents of the policy optimization objectives. Moreover, we develop a practical method that can estimate the local smoothness of objective function from samples to identify when the training process has encountered fractal landscapes. We show experiments to illustrate how some failure cases of policy optimization can be explained by such fractal landscapes.