Measuring and Mitigating Interference in Reinforcement Learning
Vincent Liu Han Wang Ruo Yu Tao Khurram Javed Adam White Martha White
Catastrophic interference is common in many network-based learning systems,
and many proposals exist for mitigating it. Before overcoming interference we
must understand it better. In this work, we provide a definition and novel
measure of interference for value-based reinforcement learning methods such as
Fitted Q-Iteration and DQN. We systematically evaluate our measure of
interference, showing that it correlates with instability in control
performance, across a variety of network architectures. Our new interference
measure allows us to ask novel scientific questions about commonly used deep
learning architectures and study learning algorithms which mitigate
interference. Lastly, we outline a class of algorithms which we call
online-aware that are designed to mitigate interference, and show they do
reduce interference according to our measure and that they improve stability
and performance in several classic control environments.