Nonlinear spiked covariance matrices and signal propagation in deep
  neural networks
Zhichao Wang Denny Wu Zhou Fan
Many recent works have studied the eigenvalue spectrum of the Conjugate
Kernel (CK) defined by the nonlinear feature map of a feedforward neural
network. However, existing results only establish weak convergence of the
empirical eigenvalue distribution, and fall short of providing precise
quantitative characterizations of the ''spike'' eigenvalues and eigenvectors
that often capture the low-dimensional signal structure of the learning
problem. In this work, we characterize these signal eigenvalues and
eigenvectors for a nonlinear version of the spiked covariance model, including
the CK as a special case. Using this general result, we give a quantitative
description of how spiked eigenstructure in the input data propagates through
the hidden layers of a neural network with random weights. As a second
application, we study a simple regime of representation learning where the
weight matrix develops a rank-one signal component over training and
characterize the alignment of the target function with the spike eigenvector of
the CK on test data.