Generative Model-based Feature Knowledge Distillation for Action
  Recognition
Guiqin Wang Peng Zhao Yanjiang Shi Cong Zhao Shusen Yang
Knowledge distillation (KD), a technique widely employed in computer vision,
has emerged as a de facto standard for improving the performance of small
neural networks. However, prevailing KD-based approaches in video tasks
primarily focus on designing loss functions and fusing cross-modal information.
This overlooks the spatial-temporal feature semantics, resulting in limited
advancements in model compression. Addressing this gap, our paper introduces an
innovative knowledge distillation framework, with the generative model for
training a lightweight student model. In particular, the framework is organized
into two steps: the initial phase is Feature Representation, wherein a
generative model-based attention module is trained to represent feature
semantics; Subsequently, the Generative-based Feature Distillation phase
encompasses both Generative Distillation and Attention Distillation, with the
objective of transferring attention-based feature semantics with the generative
model. The efficacy of our approach is demonstrated through comprehensive
experiments on diverse popular datasets, proving considerable enhancements in
video action recognition task. Moreover, the effectiveness of our proposed
framework is validated in the context of more intricate video action detection
task. Our code is available at https://github.com/aaai-24/Generative-based-KD.