RRR-Net: Reusing, Reducing, and Recycling a Deep Backbone Network
Haozhe Sun Isabelle Guyon Felix Mohr Hedi Tabia
It has become mainstream in computer vision and other machine learning
domains to reuse backbone networks pre-trained on large datasets as
preprocessors. Typically, the last layer is replaced by a shallow learning
machine of sorts; the newly-added classification head and (optionally) deeper
layers are fine-tuned on a new task. Due to its strong performance and
simplicity, a common pre-trained backbone network is ResNet152.However,
ResNet152 is relatively large and induces inference latency. In many cases, a
compact and efficient backbone with similar performance would be preferable
over a larger, slower one. This paper investigates techniques to reuse a
pre-trained backbone with the objective of creating a smaller and faster model.
Starting from a large ResNet152 backbone pre-trained on ImageNet, we first
reduce it from 51 blocks to 5 blocks, reducing its number of parameters and
FLOPs by more than 6 times, without significant performance degradation. Then,
we split the model after 3 blocks into several branches, while preserving the
same number of parameters and FLOPs, to create an ensemble of sub-networks to
improve performance. Our experiments on a large benchmark of $40$ image
classification datasets from various domains suggest that our techniques match
the performance (if not better) of ``classical backbone fine-tuning'' while
achieving a smaller model size and faster inference speed.