Multi-stage Neural Networks: Function Approximator of Machine Precision
Yongji Wang Ching-Yao Lai
Deep learning techniques are increasingly applied to scientific problems,
where the precision of networks is crucial. Despite being deemed as universal
function approximators, neural networks, in practice, struggle to reduce the
prediction errors below $O(10^{-5})$ even with large network size and extended
training iterations. To address this issue, we developed the multi-stage neural
networks that divides the training process into different stages, with each
stage using a new network that is optimized to fit the residue from the
previous stage. Across successive stages, the residue magnitudes decreases
substantially and follows an inverse power-law relationship with the residue
frequencies. The multi-stage neural networks effectively mitigate the spectral
biases associated with regular neural networks, enabling them to capture the
high frequency feature of target functions. We demonstrate that the prediction
error from the multi-stage training for both regression problems and
physics-informed neural networks can nearly reach the machine-precision
$O(10^{-16})$ of double-floating point within a finite number of iterations.
Such levels of accuracy are rarely attainable using single neural networks
alone.