A Fisher-Rao gradient flow for entropy-regularised Markov decision
  processes in Polish spaces
Bekzhan Kerimkulov James-Michael Leahy David Siska Lukasz Szpruch Yufei Zhang
We study the global convergence of a Fisher-Rao policy gradient flow for
infinite-horizon entropy-regularised Markov decision processes with Polish
state and action space. The flow is a continuous-time analogue of a policy
mirror descent method. We establish the global well-posedness of the gradient
flow and demonstrate its exponential convergence to the optimal policy.
Moreover, we prove the flow is stable with respect to gradient evaluation,
offering insights into the performance of a natural policy gradient flow with
log-linear policy parameterisation. To overcome challenges stemming from the
lack of the convexity of the objective function and the discontinuity arising
from the entropy regulariser, we leverage the performance difference lemma and
the duality relationship between the gradient and mirror descent flows.