DR-Tune: Improving Fine-tuning of Pretrained Visual Models by
  Distribution Regularization with Semantic Calibration
Nan Zhou Jiaxin Chen Di Huang
The visual models pretrained on large-scale benchmarks encode general
knowledge and prove effective in building more powerful representations for
downstream tasks. Most existing approaches follow the fine-tuning paradigm,
either by initializing or regularizing the downstream model based on the
pretrained one. The former fails to retain the knowledge in the successive
fine-tuning phase, thereby prone to be over-fitting, and the latter imposes
strong constraints to the weights or feature maps of the downstream model
without considering semantic drift, often incurring insufficient optimization.
To deal with these issues, we propose a novel fine-tuning framework, namely
distribution regularization with semantic calibration (DR-Tune). It employs
distribution regularization by enforcing the downstream task head to decrease
its classification error on the pretrained feature distribution, which prevents
it from over-fitting while enabling sufficient training of downstream encoders.
Furthermore, to alleviate the interference by semantic drift, we develop the
semantic calibration (SC) module to align the global shape and class centers of
the pretrained and downstream feature distributions. Extensive experiments on
widely used image classification datasets show that DR-Tune consistently
improves the performance when combing with various backbones under different
pretraining strategies. Code is available at:
https://github.com/weeknan/DR-Tune.