Collaboratively Self-supervised Video Representation Learning for Action
  Recognition
Jie Zhang Zhifan Wan Lanqing Hu Stephen Lin Shuzhe Wu Shiguang Shan
Considering the close connection between action recognition and human pose
estimation, we design a Collaboratively Self-supervised Video Representation
(CSVR) learning framework specific to action recognition by jointly considering
generative pose prediction and discriminative context matching as pretext
tasks. Specifically, our CSVR consists of three branches: a generative pose
prediction branch, a discriminative context matching branch, and a video
generating branch. Among them, the first one encodes dynamic motion feature by
utilizing Conditional-GAN to predict the human poses of future frames, and the
second branch extracts static context features by pulling the representations
of clips and compressed key frames from the same video together while pushing
apart the pairs from different videos. The third branch is designed to recover
the current video frames and predict the future ones, for the purpose of
collaboratively improving dynamic motion features and static context features.
Extensive experiments demonstrate that our method achieves state-of-the-art
performance on the UCF101 and HMDB51 datasets.