Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error
  Feedback
Hang Wang Sen Lin Junshan Zhang
The ensemble method is a promising way to mitigate the overestimation issue
in Q-learning, where multiple function approximators are used to estimate the
action values. It is known that the estimation bias hinges heavily on the
ensemble size (i.e., the number of Q-function approximators used in the
target), and that determining the `right' ensemble size is highly nontrivial,
because of the time-varying nature of the function approximation errors during
the learning process. To tackle this challenge, we first derive an upper bound
and a lower bound on the estimation bias, based on which the ensemble size is
adapted to drive the bias to be nearly zero, thereby coping with the impact of
the time-varying approximation errors accordingly. Motivated by the theoretic
findings, we advocate that the ensemble method can be combined with Model
Identification Adaptive Control (MIAC) for effective ensemble size adaptation.
Specifically, we devise Adaptive Ensemble Q-learning (AdaEQ), a generalized
ensemble method with two key steps: (a) approximation error characterization
which serves as the feedback for flexibly controlling the ensemble size, and
(b) ensemble size adaptation tailored towards minimizing the estimation bias.
Extensive experiments are carried out to show that AdaEQ can improve the
learning performance than the existing methods for the MuJoCo benchmark.