Never Explore Repeatedly in Multi-Agent Reinforcement Learning
Chenghao Li Tonghan Wang Chongjie Zhang Qianchuan Zhao
In the realm of multi-agent reinforcement learning, intrinsic motivations
have emerged as a pivotal tool for exploration. While the computation of many
intrinsic rewards relies on estimating variational posteriors using neural
network approximators, a notable challenge has surfaced due to the limited
expressive capability of these neural statistics approximators. We pinpoint
this challenge as the "revisitation" issue, where agents recurrently explore
confined areas of the task space. To combat this, we propose a dynamic reward
scaling approach. This method is crafted to stabilize the significant
fluctuations in intrinsic rewards in previously explored areas and promote
broader exploration, effectively curbing the revisitation phenomenon. Our
experimental findings underscore the efficacy of our approach, showcasing
enhanced performance in demanding environments like Google Research Football
and StarCraft II micromanagement tasks, especially in sparse reward settings.