Diverse Priors for Deep Reinforcement Learning
Chenfan Weng Zhongguo Li
In Reinforcement Learning (RL), agents aim at maximizing cumulative rewards
in a given environment. During the learning process, RL agents face the dilemma
of exploitation and exploration: leveraging existing knowledge to acquire
rewards or seeking potentially higher ones. Using uncertainty as a guiding
principle provides an active and effective approach to solving this dilemma and
ensemble-based methods are one of the prominent avenues for quantifying
uncertainty. Nevertheless, conventional ensemble-based uncertainty estimation
lacks an explicit prior, deviating from Bayesian principles. Besides, this
method requires diversity among members to generate less biased uncertainty
estimation results. To address the above problems, previous research has
incorporated random functions as priors. Building upon these foundational
efforts, our work introduces an innovative approach with delicately designed
prior NNs, which can incorporate maximal diversity in the initial value
functions of RL. Our method has demonstrated superior performance compared with
the random prior approaches in solving classic control problems and general
exploration tasks, significantly improving sample efficiency.