"Lossless" Compression of Deep Neural Networks: A High-dimensional
  Neural Tangent Kernel Approach
Lingyu Gu Yongqi Du Yuan Zhang Di Xie Shiliang Pu Robert C. Qiu Zhenyu Liao
Modern deep neural networks (DNNs) are extremely powerful; however, this
comes at the price of increased depth and having more parameters per layer,
making their training and inference more computationally challenging. In an
attempt to address this key limitation, efforts have been devoted to the
compression (e.g., sparsification and/or quantization) of these large-scale
machine learning models, so that they can be deployed on low-power IoT devices.
In this paper, building upon recent advances in neural tangent kernel (NTK) and
random matrix theory (RMT), we provide a novel compression approach to wide and
fully-connected \emph{deep} neural nets. Specifically, we demonstrate that in
the high-dimensional regime where the number of data points $n$ and their
dimension $p$ are both large, and under a Gaussian mixture model for the data,
there exists \emph{asymptotic spectral equivalence} between the NTK matrices
for a large family of DNN models. This theoretical result enables "lossless"
compression of a given DNN to be performed, in the sense that the compressed
network yields asymptotically the same NTK as the original (dense and
unquantized) network, with its weights and activations taking values
\emph{only} in $\{ 0, \pm 1 \}$ up to a scaling. Experiments on both synthetic
and real-world data are conducted to support the advantages of the proposed
compression scheme, with code available at
\url{https://github.com/Model-Compression/Lossless_Compression}.