A foundational neural operator that continuously learns without
  forgetting
Tapas Tripura Souvik Chakraborty
Machine learning has witnessed substantial growth, leading to the development
of advanced artificial intelligence models crafted to address a wide range of
real-world challenges spanning various domains, such as computer vision,
natural language processing, and scientific computing. Nevertheless, the
creation of custom models for each new task remains a resource-intensive
undertaking, demanding considerable computational time and memory resources. In
this study, we introduce the concept of the Neural Combinatorial Wavelet Neural
Operator (NCWNO) as a foundational model for scientific computing. This model
is specifically designed to excel in learning from a diverse spectrum of
physics and continuously adapt to the solution operators associated with
parametric partial differential equations (PDEs). The NCWNO leverages a gated
structure that employs local wavelet experts to acquire shared features across
multiple physical systems, complemented by a memory-based ensembling approach
among these local wavelet experts. This combination enables rapid adaptation to
new challenges. The proposed foundational model offers two key advantages: (i)
it can simultaneously learn solution operators for multiple parametric PDEs,
and (ii) it can swiftly generalize to new parametric PDEs with minimal
fine-tuning. The proposed NCWNO is the first foundational operator learning
algorithm distinguished by its (i) robustness against catastrophic forgetting,
(ii) the maintenance of positive transfer for new parametric PDEs, and (iii)
the facilitation of knowledge transfer across dissimilar tasks. Through an
extensive set of benchmark examples, we demonstrate that the NCWNO can
outperform task-specific baseline operator learning frameworks with minimal
hyperparameter tuning at the prediction stage. We also show that with minimal
fine-tuning, the NCWNO performs accurate combinatorial learning of new
parametric PDEs.