Leveraging Reward Consistency for Interpretable Feature Discovery in
  Reinforcement Learning
Qisen Yang Huanqian Wang Mukun Tong Wenjie Shi Gao Huang Shiji Song
The black-box nature of deep reinforcement learning (RL) hinders them from
real-world applications. Therefore, interpreting and explaining RL agents have
been active research topics in recent years. Existing methods for post-hoc
explanations usually adopt the action matching principle to enable an easy
understanding of vision-based RL agents. In this paper, it is argued that the
commonly used action matching principle is more like an explanation of deep
neural networks (DNNs) than the interpretation of RL agents. It may lead to
irrelevant or misplaced feature attribution when different DNNs' outputs lead
to the same rewards or different rewards result from the same outputs.
Therefore, we propose to consider rewards, the essential objective of RL
agents, as the essential objective of interpreting RL agents as well. To ensure
reward consistency during interpretable feature discovery, a novel framework
(RL interpreting RL, denoted as RL-in-RL) is proposed to solve the gradient
disconnection from actions to rewards. We verify and evaluate our method on the
Atari 2600 games as well as Duckietown, a challenging self-driving car
simulator environment. The results show that our method manages to keep reward
(or return) consistency and achieves high-quality feature attribution. Further,
a series of analytical experiments validate our assumption of the action
matching principle's limitations.