Deep Common Feature Mining for Efficient Video Semantic Segmentation
Yaoyan Zheng Hongyu Yang Di Huang
Recent advancements in video semantic segmentation have made substantial
progress by exploiting temporal correlations. Nevertheless, persistent
challenges, including redundant computation and the reliability of the feature
propagation process, underscore the need for further innovation. In response,
we present Deep Common Feature Mining (DCFM), a novel approach strategically
designed to address these challenges by leveraging the concept of feature
sharing. DCFM explicitly decomposes features into two complementary components.
The common representation extracted from a key-frame furnishes essential
high-level information to neighboring non-key frames, allowing for direct
re-utilization without feature propagation. Simultaneously, the independent
feature, derived from each video frame, captures rapidly changing information,
providing frame-specific clues crucial for segmentation. To achieve such
decomposition, we employ a symmetric training strategy tailored for sparsely
annotated data, empowering the backbone to learn a robust high-level
representation enriched with common information. Additionally, we incorporate a
self-supervised loss function to reinforce intra-class feature similarity and
enhance temporal consistency. Experimental evaluations on the VSPW and
Cityscapes datasets demonstrate the effectiveness of our method, showing a
superior balance between accuracy and efficiency.