No More Shortcuts: Realizing the Potential of Temporal Self-Supervision
Ishan Rajendrakumar Dave Simon Jenni Mubarak Shah
Self-supervised approaches for video have shown impressive results in video
understanding tasks. However, unlike early works that leverage temporal
self-supervision, current state-of-the-art methods primarily rely on tasks from
the image domain (e.g., contrastive learning) that do not explicitly promote
the learning of temporal features. We identify two factors that limit existing
temporal self-supervision: 1) tasks are too simple, resulting in saturated
training performance, and 2) we uncover shortcuts based on local appearance
statistics that hinder the learning of high-level features. To address these
issues, we propose 1) a more challenging reformulation of temporal
self-supervision as frame-level (rather than clip-level) recognition tasks and
2) an effective augmentation strategy to mitigate shortcuts. Our model extends
a representation of single video frames, pre-trained through contrastive
learning, with a transformer that we train through temporal self-supervision.
We demonstrate experimentally that our more challenging frame-level task
formulations and the removal of shortcuts drastically improve the quality of
features learned through temporal self-supervision. The generalization
capability of our self-supervised video method is evidenced by its
state-of-the-art performance in a wide range of high-level semantic tasks,
including video retrieval, action classification, and video attribute
recognition (such as object and scene identification), as well as low-level
temporal correspondence tasks like video object segmentation and pose tracking.
Additionally, we show that the video representations learned through our method
exhibit increased robustness to the input perturbations.