TENG: Time-Evolving Natural Gradient for Solving PDEs with Deep Neural
  Net
Zhuo Chen Jacob McCarran Esteban Vizcaino Marin Soljačić Di Luo
Partial differential equations (PDEs) are instrumental for modeling dynamical
systems in science and engineering. The advent of neural networks has initiated
a significant shift in tackling these complexities though challenges in
accuracy persist, especially for initial value problems. In this paper, we
introduce the $\textit{Time-Evolving Natural Gradient (TENG)}$, generalizing
time-dependent variational principles and optimization-based time integration,
leveraging natural gradient optimization to obtain high accuracy in
neural-network-based PDE solutions. Our comprehensive development includes
algorithms like TENG-Euler and its high-order variants, such as TENG-Heun,
tailored for enhanced precision and efficiency. TENG's effectiveness is further
validated through its performance, surpassing current leading methods and
achieving machine precision in step-by-step optimizations across a spectrum of
PDEs, including the heat equation, Allen-Cahn equation, and Burgers' equation.