Hessian Eigenvectors and Principal Component Analysis of Neural Network
  Weight Matrices
David Haink
This study delves into the intricate dynamics of trained deep neural networks
and their relationships with network parameters. Trained networks predominantly
continue training in a single direction, known as the drift mode. This drift
mode can be explained by the quadratic potential model of the loss function,
suggesting a slow exponential decay towards the potential minima. We unveil a
correlation between Hessian eigenvectors and network weights. This
relationship, hinging on the magnitude of eigenvalues, allows us to discern
parameter directions within the network. Notably, the significance of these
directions relies on two defining attributes: the curvature of their potential
wells (indicated by the magnitude of Hessian eigenvalues) and their alignment
with the weight vectors. Our exploration extends to the decomposition of weight
matrices through singular value decomposition. This approach proves practical
in identifying critical directions within the Hessian, considering both their
magnitude and curvature. Furthermore, our examination showcases the
applicability of principal component analysis in approximating the Hessian,
with update parameters emerging as a superior choice over weights for this
purpose. Remarkably, our findings unveil a similarity between the largest
Hessian eigenvalues of individual layers and the entire network. Notably,
higher eigenvalues are concentrated more in deeper layers. Leveraging these
insights, we venture into addressing catastrophic forgetting, a challenge of
neural networks when learning new tasks while retaining knowledge from previous
ones. By applying our discoveries, we formulate an effective strategy to
mitigate catastrophic forgetting, offering a possible solution that can be
applied to networks of varying scales, including larger architectures.