Exponential Expressivity of ReLU$^k$ Neural Networks on Gevrey Classes
  with Point Singularities
Joost A. A. Opschoor Christoph Schwab
We analyze deep Neural Network emulation rates of smooth functions with point
singularities in bounded, polytopal domains $\mathrm{D} \subset \mathbb{R}^d$,
$d=2,3$. We prove exponential emulation rates in Sobolev spaces in terms of the
number of neurons and in terms of the number of nonzero coefficients for
Gevrey-regular solution classes defined in terms of weighted Sobolev scales in
$\mathrm{D}$, comprising the countably-normed spaces of I.M. Babu\v{s}ka and
B.Q. Guo.
  As intermediate result, we prove that continuous, piecewise polynomial high
order (``$p$-version'') finite elements with elementwise polynomial degree
$p\in\mathbb{N}$ on arbitrary, regular, simplicial partitions of polyhedral
domains $\mathrm{D} \subset \mathbb{R}^d$, $d\geq 2$ can be exactly emulated by
neural networks combining ReLU and ReLU$^2$ activations. On shape-regular,
simplicial partitions of polytopal domains $\mathrm{D}$, both the number of
neurons and the number of nonzero parameters are proportional to the number of
degrees of freedom of the finite element space, in particular for the
$hp$-Finite Element Method of I.M. Babu\v{s}ka and B.Q. Guo.