Deep ReLU networks and high-order finite element methods II: Chebyshev
  emulation
Joost A. A. Opschoor Christoph Schwab
Expression rates and stability in Sobolev norms of deep ReLU neural networks
(NNs) in terms of the number of parameters defining the NN for continuous,
piecewise polynomial functions, on arbitrary, finite partitions $\mathcal{T}$
of a bounded interval $(a,b)$ are addressed. Novel constructions of ReLU NN
surrogates encoding the approximated functions in terms of Chebyshev polynomial
expansion coefficients are developed. Chebyshev coefficients can be computed
easily from the values of the function in the Clenshaw--Curtis points using the
inverse fast Fourier transform. Bounds on expression rates and stability that
are superior to those of constructions based on ReLU NN emulations of monomials
considered in [Opschoor, Petersen, Schwab, 2020] are obtained. All emulation
bounds are explicit in terms of the (arbitrary) partition of the interval, the
target emulation accuracy and the polynomial degree in each element of the
partition. ReLU NN emulation error estimates are provided for various classes
of functions and norms, commonly encountered in numerical analysis. In
particular, we show exponential ReLU emulation rate bounds for analytic
functions with point singularities and develop an interface between Chebfun
approximations and constructive ReLU NN emulations.