Understanding Your Agent: Leveraging Large Language Models for Behavior
  Explanation
Xijia Zhang Yue Guo Simon Stepputtis Katia Sycara Joseph Campbell
Intelligent agents such as robots are increasingly deployed in real-world,
safety-critical settings. It is vital that these agents are able to explain the
reasoning behind their decisions to human counterparts; however, their behavior
is often produced by uninterpretable models such as deep neural networks. We
propose an approach to generate natural language explanations for an agent's
behavior based only on observations of states and actions, thus making our
method independent from the underlying model's representation. For such models,
we first learn a behavior representation and subsequently use it to produce
plausible explanations with minimal hallucination while affording user
interaction with a pre-trained large language model. We evaluate our method in
a multi-agent search-and-rescue environment and demonstrate the effectiveness
of our explanations for agents executing various behaviors. Through user
studies and empirical experiments, we show that our approach generates
explanations as helpful as those produced by a human domain expert while
enabling beneficial interactions such as clarification and counterfactual
queries.