Revisiting Experience Replayable Conditions
Taisuke Kobayashi
Experience replay (ER) used in (deep) reinforcement learning is considered to
be applicable only to off-policy algorithms. However, there have been some
cases in which ER has been applied for on-policy algorithms, suggesting that
off-policyness might be a sufficient condition for applying ER. This paper
reconsiders more strict "experience replayable conditions" (ERC) and proposes
the way of modifying the existing algorithms to satisfy ERC. To this end,
instability of policy improvements is assumed to be a key in ERC. The
instability factors are revealed from the viewpoint of metric learning as i)
repulsive forces from negative samples and ii) replays of inappropriate
experiences. Accordingly, the corresponding stabilization tricks are derived.
As a result, it is confirmed through numerical simulations that the proposed
stabilization tricks make ER applicable to an advantage actor-critic, an
on-policy algorithm. In addition, its learning performance is comparable to
that of a soft actor-critic, a state-of-the-art off-policy algorithm.