Neural Networks for Singular Perturbations
Joost A. A. Opschoor Christoph Schwab Christos Xenophontos
We prove deep neural network (DNN for short) expressivity rate bounds for
solution sets of a model class of singularly perturbed, elliptic two-point
boundary value problems, in Sobolev norms, on the bounded interval $(-1,1)$. We
assume that the given source term and reaction coefficient are analytic in
$[-1,1]$.
  We establish expression rate bounds in Sobolev norms in terms of the NN size
which are uniform with respect to the singular perturbation parameter for
several classes of DNN architectures. In particular, ReLU NNs, spiking NNs, and
$\tanh$- and sigmoid-activated NNs. The latter activations can represent
``exponential boundary layer solution features'' explicitly, in the last hidden
layer of the DNN, i.e. in a shallow subnetwork, and afford improved robust
expression rate bounds in terms of the NN size.
  We prove that all DNN architectures allow robust exponential solution
expression in so-called `energy' as well as in `balanced' Sobolev norms, for
analytic input data.