Modify Training Directions in Function Space to Reduce Generalization
  Error
Yi Yu Wenlian Lu Boyu Chen
We propose theoretical analyses of a modified natural gradient descent method
in the neural network function space based on the eigendecompositions of neural
tangent kernel and Fisher information matrix. We firstly present analytical
expression for the function learned by this modified natural gradient under the
assumptions of Gaussian distribution and infinite width limit. Thus, we
explicitly derive the generalization error of the learned neural network
function using theoretical methods from eigendecomposition and statistics
theory. By decomposing of the total generalization error attributed to
different eigenspace of the kernel in function space, we propose a criterion
for balancing the errors stemming from training set and the distribution
discrepancy between the training set and the true data. Through this approach,
we establish that modifying the training direction of the neural network in
function space leads to a reduction in the total generalization error.
Furthermore, We demonstrate that this theoretical framework is capable to
explain many existing results of generalization enhancing methods. These
theoretical results are also illustrated by numerical examples on synthetic
data.