M$^3$Net: Multi-view Encoding, Matching, and Fusion for Few-shot
  Fine-grained Action Recognition
Hao Tang Jun Liu Shuanglin Yan Rui Yan Zechao Li Jinhui Tang
Due to the scarcity of manually annotated data required for fine-grained
video understanding, few-shot fine-grained (FS-FG) action recognition has
gained significant attention, with the aim of classifying novel fine-grained
action categories with only a few labeled instances. Despite the progress made
in FS coarse-grained action recognition, current approaches encounter two
challenges when dealing with the fine-grained action categories: the inability
to capture subtle action details and the insufficiency of learning from limited
data that exhibit high intra-class variance and inter-class similarity. To
address these limitations, we propose M$^3$Net, a matching-based framework for
FS-FG action recognition, which incorporates \textit{multi-view encoding},
\textit{multi-view matching}, and \textit{multi-view fusion} to facilitate
embedding encoding, similarity matching, and decision making across multiple
viewpoints. \textit{Multi-view encoding} captures rich contextual details from
the intra-frame, intra-video, and intra-episode perspectives, generating
customized higher-order embeddings for fine-grained data. \textit{Multi-view
matching} integrates various matching functions enabling flexible relation
modeling within limited samples to handle multi-scale spatio-temporal
variations by leveraging the instance-specific, category-specific, and
task-specific perspectives. \textit{Multi-view fusion} consists of
matching-predictions fusion and matching-losses fusion over the above views,
where the former promotes mutual complementarity and the latter enhances
embedding generalizability by employing multi-task collaborative learning.
Explainable visualizations and experimental results on three challenging
benchmarks demonstrate the superiority of M$^3$Net in capturing fine-grained
action details and achieving state-of-the-art performance for FS-FG action
recognition.