Fractal Landscapes in Policy Optimization
Tao Wang Sylvia Herbert Sicun Gao
Policy gradient lies at the core of deep reinforcement learning (RL) in
continuous domains. Despite much success, it is often observed in practice that
RL training with policy gradient can fail for many reasons, even on standard
control problems with known solutions. We propose a framework for understanding
one inherent limitation of the policy gradient approach: the optimization
landscape in the policy space can be extremely non-smooth or fractal for
certain classes of MDPs, such that there does not exist gradient to be
estimated in the first place. We draw on techniques from chaos theory and
non-smooth analysis, and analyze the maximal Lyapunov exponents and H\"older
exponents of the policy optimization objectives. Moreover, we develop a
practical method that can estimate the local smoothness of objective function
from samples to identify when the training process has encountered fractal
landscapes. We show experiments to illustrate how some failure cases of policy
optimization can be explained by such fractal landscapes.