Closing the ODE-SDE gap in score-based diffusion models through the
  Fokker-Planck equation
Teo Deveney Jan Stanczuk Lisa Maria Kreusser Chris Budd Carola-Bibiane Schönlieb
Score-based diffusion models have emerged as one of the most promising
frameworks for deep generative modelling, due to their state-of-the art
performance in many generation tasks while relying on mathematical foundations
such as stochastic differential equations (SDEs) and ordinary differential
equations (ODEs). Empirically, it has been reported that ODE based samples are
inferior to SDE based samples. In this paper we rigorously describe the range
of dynamics and approximations that arise when training score-based diffusion
models, including the true SDE dynamics, the neural approximations, the various
approximate particle dynamics that result, as well as their associated
Fokker--Planck equations and the neural network approximations of these
Fokker--Planck equations. We systematically analyse the difference between the
ODE and SDE dynamics of score-based diffusion models, and link it to an
associated Fokker--Planck equation. We derive a theoretical upper bound on the
Wasserstein 2-distance between the ODE- and SDE-induced distributions in terms
of a Fokker--Planck residual. We also show numerically that conventional
score-based diffusion models can exhibit significant differences between ODE-
and SDE-induced distributions which we demonstrate using explicit comparisons.
Moreover, we show numerically that reducing the Fokker--Planck residual by
adding it as an additional regularisation term leads to closing the gap between
ODE- and SDE-induced distributions. Our experiments suggest that this
regularisation can improve the distribution generated by the ODE, however that
this can come at the cost of degraded SDE sample quality.