Hierarchical Task Network Planning for Facilitating Cooperative
  Multi-Agent Reinforcement Learning
Xuechen Mu Hankz Hankui Zhuo Chen Chen Kai Zhang Chao Yu Jianye Hao
Exploring sparse reward multi-agent reinforcement learning (MARL)
environments with traps in a collaborative manner is a complex task. Agents
typically fail to reach the goal state and fall into traps, which affects the
overall performance of the system. To overcome this issue, we present SOMARL, a
framework that uses prior knowledge to reduce the exploration space and assist
learning. In SOMARL, agents are treated as part of the MARL environment, and
symbolic knowledge is embedded using a tree structure to build a knowledge
hierarchy. The framework has a two-layer hierarchical structure, comprising a
hybrid module with a Hierarchical Task Network (HTN) planning and
meta-controller at the higher level, and a MARL-based interactive module at the
lower level. The HTN module and meta-controller use Hierarchical Domain
Definition Language (HDDL) and the option framework to formalize symbolic
knowledge and obtain domain knowledge and a symbolic option set, respectively.
Moreover, the HTN module leverages domain knowledge to guide low-level agent
exploration by assisting the meta-controller in selecting symbolic options. The
meta-controller further computes intrinsic rewards of symbolic options to limit
exploration behavior and adjust HTN planning solutions as needed. We evaluate
SOMARL on two benchmarks, FindTreasure and MoveBox, and report superior
performance over state-of-the-art MARL and subgoal-based baselines for MARL
environments significantly.