Measuring Exploration in Reinforcement Learning via Optimal Transport in
  Policy Space
Reabetswe M. Nkhumise Debabrota Basu Tony J. Prescott Aditya Gilra
Exploration is the key ingredient of reinforcement learning (RL) that
determines the speed and success of learning. Here, we quantify and compare the
amount of exploration and learning accomplished by a Reinforcement Learning
(RL) algorithm. Specifically, we propose a novel measure, named Exploration
Index, that quantifies the relative effort of knowledge transfer
(transferability) by an RL algorithm in comparison to supervised learning (SL)
that transforms the initial data distribution of RL to the corresponding final
data distribution. The comparison is established by formulating learning in RL
as a sequence of SL tasks, and using optimal transport based metrics to compare
the total path traversed by the RL and SL algorithms in the data distribution
space. We perform extensive empirical analysis on various environments and with
multiple algorithms to demonstrate that the exploration index yields insights
about the exploration behaviour of any RL algorithm, and also allows us to
compare the exploratory behaviours of different RL algorithms.