Learning Diverse Policies with Soft Self-Generated Guidance
Guojian Wang Faguo Wu Xiao Zhang Jianxiang Liu
Reinforcement learning (RL) with sparse and deceptive rewards is challenging
because non-zero rewards are rarely obtained. Hence, the gradient calculated by
the agent can be stochastic and without valid information. Recent studies that
utilize memory buffers of previous experiences can lead to a more efficient
learning process. However, existing methods often require these experiences to
be successful and may overly exploit them, which can cause the agent to adopt
suboptimal behaviors. This paper develops an approach that uses diverse past
trajectories for faster and more efficient online RL, even if these
trajectories are suboptimal or not highly rewarded. The proposed algorithm
combines a policy improvement step with an additional exploration step using
offline demonstration data. The main contribution of this paper is that by
regarding diverse past trajectories as guidance, instead of imitating them, our
method directs its policy to follow and expand past trajectories while still
being able to learn without rewards and approach optimality. Furthermore, a
novel diversity measurement is introduced to maintain the team's diversity and
regulate exploration. The proposed algorithm is evaluated on discrete and
continuous control tasks with sparse and deceptive rewards. Compared with the
existing RL methods, the experimental results indicate that our proposed
algorithm is significantly better than the baseline methods regarding diverse
exploration and avoiding local optima.