Is Exploration All You Need? Effective Exploration Characteristics for
  Transfer in Reinforcement Learning
Jonathan C. Balloch Rishav Bhagat Geigh Zollicoffer Ruoran Jia Julia Kim Mark O. Riedl
In deep reinforcement learning (RL) research, there has been a concerted
effort to design more efficient and productive exploration methods while
solving sparse-reward problems. These exploration methods often share common
principles (e.g., improving diversity) and implementation details (e.g.,
intrinsic reward). Prior work found that non-stationary Markov decision
processes (MDPs) require exploration to efficiently adapt to changes in the
environment with online transfer learning. However, the relationship between
specific exploration characteristics and effective transfer learning in deep RL
has not been characterized. In this work, we seek to understand the
relationships between salient exploration characteristics and improved
performance and efficiency in transfer learning. We test eleven popular
exploration algorithms on a variety of transfer types -- or ``novelties'' -- to
identify the characteristics that positively affect online transfer learning.
Our analysis shows that some characteristics correlate with improved
performance and efficiency across a wide range of transfer tasks, while others
only improve transfer performance with respect to specific environment changes.
From our analysis, make recommendations about which exploration algorithm
characteristics are best suited to specific transfer situations.