Adapting Double Q-Learning for Continuous Reinforcement Learning
Arsenii Kuznetsov
Majority of off-policy reinforcement learning algorithms use overestimation
bias control techniques. Most of these techniques rooted in heuristics,
primarily addressing the consequences of overestimation rather than its
fundamental origins. In this work we present a novel approach to the bias
correction, similar in spirit to Double Q-Learning. We propose using a policy
in form of a mixture with two components. Each policy component is maximized
and assessed by separate networks, which removes any basis for the
overestimation bias. Our approach shows promising near-SOTA results on a small
set of MuJoCo environments.