Few-shot Action Recognition via Intra- and Inter-Video Information
  Maximization
Huabin Liu Weiyao Lin Tieyuan Chen Yuxi Li Shuyuan Li John See
Current few-shot action recognition involves two primary sources of
information for classification:(1) intra-video information, determined by frame
content within a single video clip, and (2) inter-video information, measured
by relationships (e.g., feature similarity) among videos. However, existing
methods inadequately exploit these two information sources. In terms of
intra-video information, current sampling operations for input videos may omit
critical action information, reducing the utilization efficiency of video data.
For the inter-video information, the action misalignment among videos makes it
challenging to calculate precise relationships. Moreover, how to jointly
consider both inter- and intra-video information remains under-explored for
few-shot action recognition. To this end, we propose a novel framework, Video
Information Maximization (VIM), for few-shot video action recognition. VIM is
equipped with an adaptive spatial-temporal video sampler and a spatiotemporal
action alignment model to maximize intra- and inter-video information,
respectively. The video sampler adaptively selects important frames and
amplifies critical spatial regions for each input video based on the task at
hand. This preserves and emphasizes informative parts of video clips while
eliminating interference at the data level. The alignment model performs
temporal and spatial action alignment sequentially at the feature level,
leading to more precise measurements of inter-video similarity. Finally, These
goals are facilitated by incorporating additional loss terms based on mutual
information measurement. Consequently, VIM acts to maximize the distinctiveness
of video information from limited video data. Extensive experimental results on
public datasets for few-shot action recognition demonstrate the effectiveness
and benefits of our framework.