Regression-Oriented Knowledge Distillation for Lightweight Ship
  Orientation Angle Prediction with Optical Remote Sensing Images
Zhan Shi Xin Ding Peng Ding Chun Yang Ru Huang Xiaoxuan Song
Ship orientation angle prediction (SOAP) with optical remote sensing images
is an important image processing task, which often relies on deep convolutional
neural networks (CNNs) to make accurate predictions. This paper proposes a
novel framework to reduce the model sizes and computational costs of SOAP
models without harming prediction accuracy. First, a new SOAP model called
Mobile-SOAP is designed based on MobileNetV2, achieving state-of-the-art
prediction accuracy. Four tiny SOAP models are also created by replacing the
convolutional blocks in Mobile-SOAP with four small-scale networks,
respectively. Then, to transfer knowledge from Mobile-SOAP to four lightweight
models, we propose a novel knowledge distillation (KD) framework termed SOAP-KD
consisting of a novel feature-based guidance loss and an optimized synthetic
samples-based knowledge transfer mechanism. Lastly, extensive experiments on
the FGSC-23 dataset confirm the superiority of Mobile-SOAP over existing models
and also demonstrate the effectiveness of SOAP-KD in improving the prediction
performance of four specially designed tiny models. Notably, by using SOAP-KD,
the test mean absolute error of the ShuffleNetV2x1.0-based model is only 8%
higher than that of Mobile-SOAP, but its number of parameters and
multiply-accumulate operations (MACs) are respectively 61.6% and 60.8% less.