Rates of Approximation by ReLU Shallow Neural Networks
Tong Mao Ding-Xuan Zhou
Neural networks activated by the rectified linear unit (ReLU) play a central
role in the recent development of deep learning. The topic of approximating
functions from H\"older spaces by these networks is crucial for understanding
the efficiency of the induced learning algorithms. Although the topic has been
well investigated in the setting of deep neural networks with many layers of
hidden neurons, it is still open for shallow networks having only one hidden
layer. In this paper, we provide rates of uniform approximation by these
networks. We show that ReLU shallow neural networks with $m$ hidden neurons can
uniformly approximate functions from the H\"older space $W_\infty^r([-1, 1]^d)$
with rates $O((\log m)^{\frac{1}{2} +d}m^{-\frac{r}{d}\frac{d+2}{d+4}})$ when
$r<d/2 +2$. Such rates are very close to the optimal one $O(m^{-\frac{r}{d}})$
in the sense that $\frac{d+2}{d+4}$ is close to $1$, when the dimension $d$ is
large.