Operator Learning Meets Numerical Analysis: Improving Neural Networks
  through Iterative Methods
Emanuele Zappala Daniel Levine Sizhuang He Syed Rizvi Sacha Levy David van Dijk
Deep neural networks, despite their success in numerous applications, often
function without established theoretical foundations. In this paper, we bridge
this gap by drawing parallels between deep learning and classical numerical
analysis. By framing neural networks as operators with fixed points
representing desired solutions, we develop a theoretical framework grounded in
iterative methods for operator equations. Under defined conditions, we present
convergence proofs based on fixed point theory. We demonstrate that popular
architectures, such as diffusion models and AlphaFold, inherently employ
iterative operator learning. Empirical assessments highlight that performing
iterations through network operators improves performance. We also introduce an
iterative graph neural network, PIGN, that further demonstrates benefits of
iterations. Our work aims to enhance the understanding of deep learning by
merging insights from numerical analysis, potentially guiding the design of
future networks with clearer theoretical underpinnings and improved
performance.