Behind the Myth of Exploration in Policy Gradients
Adrien Bolland Gaspard Lambrechts Damien Ernst
Policy-gradient algorithms are effective reinforcement learning methods for
solving control problems with continuous state and action spaces. To compute
near-optimal policies, it is essential in practice to include exploration terms
in the learning objective. Although the effectiveness of these terms is usually
justified by an intrinsic need to explore environments, we propose a novel
analysis and distinguish two different implications of these techniques. First,
they make it possible to smooth the learning objective and to eliminate local
optima while preserving the global maximum. Second, they modify the gradient
estimates, increasing the probability that the stochastic parameter update
eventually provides an optimal policy. In light of these effects, we discuss
and illustrate empirically exploration strategies based on entropy bonuses,
highlighting their limitations and opening avenues for future works in the
design and analysis of such strategies.