ELRT: Efficient Low-Rank Training for Compact Convolutional Neural
  Networks
Yang Sui Miao Yin Yu Gong Jinqi Xiao Huy Phan Bo Yuan
Low-rank compression, a popular model compression technique that produces
compact convolutional neural networks (CNNs) with low rankness, has been
well-studied in the literature. On the other hand, low-rank training, as an
alternative way to train low-rank CNNs from scratch, has been exploited little
yet. Unlike low-rank compression, low-rank training does not need pre-trained
full-rank models, and the entire training phase is always performed on the
low-rank structure, bringing attractive benefits for practical applications.
However, the existing low-rank training solutions still face several
challenges, such as a considerable accuracy drop and/or still needing to update
full-size models during the training. In this paper, we perform a systematic
investigation on low-rank CNN training. By identifying the proper low-rank
format and performance-improving strategy, we propose ELRT, an efficient
low-rank training solution for high-accuracy, high-compactness, low-rank CNN
models. Our extensive evaluation results for training various CNNs on different
datasets demonstrate the effectiveness of ELRT.