Semantic-aware Video Representation for Few-shot Action Recognition
Yutao Tang Benjamin Bejar Rene Vidal
Recent work on action recognition leverages 3D features and textual
information to achieve state-of-the-art performance. However, most of the
current few-shot action recognition methods still rely on 2D frame-level
representations, often require additional components to model temporal
relations, and employ complex distance functions to achieve accurate alignment
of these representations. In addition, existing methods struggle to effectively
integrate textual semantics, some resorting to concatenation or addition of
textual and visual features, and some using text merely as an additional
supervision without truly achieving feature fusion and information transfer
from different modalities. In this work, we propose a simple yet effective
Semantic-Aware Few-Shot Action Recognition (SAFSAR) model to address these
issues. We show that directly leveraging a 3D feature extractor combined with
an effective feature-fusion scheme, and a simple cosine similarity for
classification can yield better performance without the need of extra
components for temporal modeling or complex distance functions. We introduce an
innovative scheme to encode the textual semantics into the video representation
which adaptively fuses features from text and video, and encourages the visual
encoder to extract more semantically consistent features. In this scheme,
SAFSAR achieves alignment and fusion in a compact way. Experiments on five
challenging few-shot action recognition benchmarks under various settings
demonstrate that the proposed SAFSAR model significantly improves the
state-of-the-art performance.