The Challenges of the Nonlinear Regime for Physics-Informed Neural
  Networks
Andrea Bonfanti Giuseppe Bruno Cristina Cipriani
The Neural Tangent Kernel (NTK) viewpoint represents a valuable approach to
examine the training dynamics of Physics-Informed Neural Networks (PINNs) in
the infinite width limit. We leverage this perspective and focus on the case of
nonlinear Partial Differential Equations (PDEs) solved by PINNs. We provide
theoretical results on the different behaviors of the NTK depending on the
linearity of the differential operator. Moreover, inspired by our theoretical
results, we emphasize the advantage of employing second-order methods for
training PINNs. Additionally, we explore the convergence capabilities of
second-order methods and address the challenges of spectral bias and slow
convergence. Every theoretical result is supported by numerical examples with
both linear and nonlinear PDEs, and we validate our training method on
benchmark test cases.