An operator preconditioning perspective on training in physics-informed
  machine learning
Tim De Ryck Florent Bonnet Siddhartha Mishra Emmanuel de Bézenac
In this paper, we investigate the behavior of gradient descent algorithms in
physics-informed machine learning methods like PINNs, which minimize residuals
connected to partial differential equations (PDEs). Our key result is that the
difficulty in training these models is closely related to the conditioning of a
specific differential operator. This operator, in turn, is associated to the
Hermitian square of the differential operator of the underlying PDE. If this
operator is ill-conditioned, it results in slow or infeasible training.
Therefore, preconditioning this operator is crucial. We employ both rigorous
mathematical analysis and empirical evaluations to investigate various
strategies, explaining how they better condition this critical operator, and
consequently improve training.