Bias-Variance Trade-off in Physics-Informed Neural Networks with
  Randomized Smoothing for High-Dimensional PDEs
Zheyuan Hu Zhouhao Yang Yezhen Wang George Em Karniadakis Kenji Kawaguchi
While physics-informed neural networks (PINNs) have been proven effective for
low-dimensional partial differential equations (PDEs), the computational cost
remains a hurdle in high-dimensional scenarios. This is particularly pronounced
when computing high-order and high-dimensional derivatives in the
physics-informed loss. Randomized Smoothing PINN (RS-PINN) introduces Gaussian
noise for stochastic smoothing of the original neural net model, enabling Monte
Carlo methods for derivative approximation, eliminating the need for costly
auto-differentiation. Despite its computational efficiency in high dimensions,
RS-PINN introduces biases in both loss and gradients, negatively impacting
convergence, especially when coupled with stochastic gradient descent (SGD). We
present a comprehensive analysis of biases in RS-PINN, attributing them to the
nonlinearity of the Mean Squared Error (MSE) loss and the PDE nonlinearity. We
propose tailored bias correction techniques based on the order of PDE
nonlinearity. The unbiased RS-PINN allows for a detailed examination of its
pros and cons compared to the biased version. Specifically, the biased version
has a lower variance and runs faster than the unbiased version, but it is less
accurate due to the bias. To optimize the bias-variance trade-off, we combine
the two approaches in a hybrid method that balances the rapid convergence of
the biased version with the high accuracy of the unbiased version. In addition,
we present an enhanced implementation of RS-PINN. Extensive experiments on
diverse high-dimensional PDEs, including Fokker-Planck, HJB, viscous Burgers',
Allen-Cahn, and Sine-Gordon equations, illustrate the bias-variance trade-off
and highlight the effectiveness of the hybrid RS-PINN. Empirical guidelines are
provided for selecting biased, unbiased, or hybrid versions, depending on the
dimensionality and nonlinearity of the specific PDE problem.