Few-shot Action Recognition with Captioning Foundation Models
Xiang Wang Shiwei Zhang Hangjie Yuan Yingya Zhang Changxin Gao Deli Zhao Nong Sang
Transferring vision-language knowledge from pretrained multimodal foundation
models to various downstream tasks is a promising direction. However, most
current few-shot action recognition methods are still limited to a single
visual modality input due to the high cost of annotating additional textual
descriptions. In this paper, we develop an effective plug-and-play framework
called CapFSAR to exploit the knowledge of multimodal models without manually
annotating text. To be specific, we first utilize a captioning foundation model
(i.e., BLIP) to extract visual features and automatically generate associated
captions for input videos. Then, we apply a text encoder to the synthetic
captions to obtain representative text embeddings. Finally, a visual-text
aggregation module based on Transformer is further designed to incorporate
cross-modal spatio-temporal complementary information for reliable few-shot
matching. In this way, CapFSAR can benefit from powerful multimodal knowledge
of pretrained foundation models, yielding more comprehensive classification in
the low-shot regime. Extensive experiments on multiple standard few-shot
benchmarks demonstrate that the proposed CapFSAR performs favorably against
existing methods and achieves state-of-the-art performance. The code will be
made publicly available.