Reusing Pretrained Models by Multi-linear Operators for Efficient
  Training
Yu Pan Ye Yuan Yichun Yin Zenglin Xu Lifeng Shang Xin Jiang Qun Liu
Training large models from scratch usually costs a substantial amount of
resources. Towards this problem, recent studies such as bert2BERT and LiGO have
reused small pretrained models to initialize a large model (termed the ``target
model''), leading to a considerable acceleration in training. Despite the
successes of these previous studies, they grew pretrained models by mapping
partial weights only, ignoring potential correlations across the entire model.
As we show in this paper, there are inter- and intra-interactions among the
weights of both the pretrained and the target models. As a result, the partial
mapping may not capture the complete information and lead to inadequate growth.
In this paper, we propose a method that linearly correlates each weight of the
target model to all the weights of the pretrained model to further enhance
acceleration ability. We utilize multi-linear operators to reduce computational
and spacial complexity, enabling acceptable resource requirements. Experiments
demonstrate that our method can save 76\% computational costs on DeiT-base
transferred from DeiT-small, which outperforms bert2BERT by +12.0\% and LiGO by
+20.7\%, respectively.