Collaborative Weakly Supervised Video Correlation Learning for
  Procedure-Aware Instructional Video Analysis
Tianyao He Huabin Liu Yuxi Li Xiao Ma Cheng Zhong Yang Zhang Weiyao Lin
Video Correlation Learning (VCL), which aims to analyze the relationships
between videos, has been widely studied and applied in various general video
tasks. However, applying VCL to instructional videos is still quite challenging
due to their intrinsic procedural temporal structure. Specifically, procedural
knowledge is critical for accurate correlation analyses on instructional
videos. Nevertheless, current procedure-learning methods heavily rely on
step-level annotations, which are costly and not scalable. To address this
problem, we introduce a weakly supervised framework called Collaborative
Procedure Alignment (CPA) for procedure-aware correlation learning on
instructional videos. Our framework comprises two core modules: collaborative
step mining and frame-to-step alignment. The collaborative step mining module
enables simultaneous and consistent step segmentation for paired videos,
leveraging the semantic and temporal similarity between frames. Based on the
identified steps, the frame-to-step alignment module performs alignment between
the frames and steps across videos. The alignment result serves as a
measurement of the correlation distance between two videos. We instantiate our
framework in two distinct instructional video tasks: sequence verification and
action quality assessment. Extensive experiments validate the effectiveness of
our approach in providing accurate and interpretable correlation analyses for
instructional videos.