A Minimal Control Family of Dynamical Syetem for Universal Approximation
Yifei Duan Yongqiang Cai
The universal approximation property (UAP) of neural networks is a
fundamental characteristic of deep learning. It is widely recognized that a
composition of linear functions and non-linear functions, such as the rectified
linear unit (ReLU) activation function, can approximate continuous functions on
compact domains. In this paper, we extend this efficacy to the scenario of
dynamical systems with controls. We prove that the control family
$\mathcal{F}_1 = \mathcal{F}_0 \cup \{ \text{ReLU}(\cdot)\} $ is enough to
generate flow maps that can uniformly approximate diffeomorphisms of
$\mathbb{R}^d$ on any compact domain, where $\mathcal{F}_0 = \{x \mapsto Ax+b:
A\in \mathbb{R}^{d\times d}, b \in \mathbb{R}^d\}$ is the set of linear maps
and the dimension $d\ge2$. Since $\mathcal{F}_1$ contains only one nonlinear
function and $\mathcal{F}_0$ does not hold the UAP, we call $\mathcal{F}_1$ a
minimal control family for UAP. Based on this, some sufficient conditions, such
as the affine invariance, on the control family are established and discussed.
Our result reveals an underlying connection between the approximation power of
neural networks and control systems.