LoSA: Long-Short-range Adapter for Scaling End-to-End Temporal Action
  Localization
Akshita Gupta Gaurav Mittal Ahmed Magooda Ye Yu Graham W. Taylor Mei Chen
Temporal Action Localization (TAL) involves localizing and classifying action
snippets in an untrimmed video. The emergence of large video foundation models
has led RGB-only video backbones to outperform previous methods needing both
RGB and optical flow modalities. Leveraging these large models is often limited
to training only the TAL head due to the prohibitively large GPU memory
required to adapt the video backbone for TAL. To overcome this limitation, we
introduce LoSA, the first memory-and-parameter-efficient backbone adapter
designed specifically for TAL to handle untrimmed videos. LoSA specializes for
TAL by introducing Long-Short-range Adapters that adapt the intermediate layers
of the video backbone over different temporal ranges. These adapters run
parallel to the video backbone to significantly reduce memory footprint. LoSA
also includes Long-Short-range Fusion that strategically combines the output of
these adapters from the video backbone layers to enhance the video features
provided to the TAL head. Experiments show that LoSA significantly outperforms
all existing methods on standard TAL benchmarks, THUMOS-14 and
ActivityNet-v1.3, by scaling end-to-end backbone adaptation to
billion-parameter-plus models like VideoMAEv2~(ViT-g) and leveraging them
beyond head-only transfer learning.