On the Optimal Expressive Power of ReLU DNNs and Its Application in
  Approximation with Kolmogorov Superposition Theorem
Juncai He
This paper is devoted to studying the optimal expressive power of ReLU deep
neural networks (DNNs) and its application in approximation via the Kolmogorov
Superposition Theorem. We first constructively prove that any continuous
piecewise linear functions on $[0,1]$, comprising $O(N^2L)$ segments, can be
represented by ReLU DNNs with $L$ hidden layers and $N$ neurons per layer.
Subsequently, we demonstrate that this construction is optimal regarding the
parameter count of the DNNs, achieved through investigating the shattering
capacity of ReLU DNNs. Moreover, by invoking the Kolmogorov Superposition
Theorem, we achieve an enhanced approximation rate for ReLU DNNs of arbitrary
width and depth when dealing with continuous functions in high-dimensional
spaces.