Multiscale Memory Comparator Transformer for Few-Shot Video Segmentation
Mennatullah Siam Rezaul Karim He Zhao Richard Wildes
Few-shot video segmentation is the task of delineating a specific novel class
in a query video using few labelled support images. Typical approaches compare
support and query features while limiting comparisons to a single feature layer
and thereby ignore potentially valuable information. We present a meta-learned
Multiscale Memory Comparator (MMC) for few-shot video segmentation that
combines information across scales within a transformer decoder. Typical
multiscale transformer decoders for segmentation tasks learn a compressed
representation, their queries, through information exchange across scales.
Unlike previous work, we instead preserve the detailed feature maps during
across scale information exchange via a multiscale memory transformer decoding
to reduce confusion between the background and novel class. Integral to the
approach, we investigate multiple forms of information exchange across scales
in different tasks and provide insights with empirical evidence on which to use
in each task. The overall comparisons among query and support features benefit
from both rich semantics and precise localization. We demonstrate our approach
primarily on few-shot video object segmentation and an adapted version on the
fully supervised counterpart. In all cases, our approach outperforms the
baseline and yields state-of-the-art performance. Our code is publicly
available at https://github.com/MSiam/MMC-MultiscaleMemory.