The kernel-balanced equation for deep neural networks
Kenichi Nakazato
Deep neural networks have shown many fruitful applications in this decade. A
network can get the generalized function through training with a finite
dataset. The degree of generalization is a realization of the proximity scale
in the data space. Specifically, the scale is not clear if the dataset is
complicated. Here we consider a network for the distribution estimation of the
dataset. We show the estimation is unstable and the instability depends on the
data density and training duration. We derive the kernel-balanced equation,
which gives a short phenomenological description of the solution. The equation
tells us the reason for the instability and the mechanism of the scale. The
network outputs a local average of the dataset as a prediction and the scale of
averaging is determined along the equation. The scale gradually decreases along
training and finally results in instability in our case.