Propagating Semantic Labels in Video Data
David Balaban Justin Medich Pranay Gosar Justin Hart
Semantic Segmentation combines two sub-tasks: the identification of
pixel-level image masks and the application of semantic labels to those masks.
Recently, so-called Foundation Models have been introduced; general models
trained on very large datasets which can be specialized and applied to more
specific tasks. One such model, the Segment Anything Model (SAM), performs
image segmentation. Semantic segmentation systems such as CLIPSeg and MaskRCNN
are trained on datasets of paired segments and semantic labels. Manual labeling
of custom data, however, is time-consuming. This work presents a method for
performing segmentation for objects in video. Once an object has been found in
a frame of video, the segment can then be propagated to future frames; thus
reducing manual annotation effort. The method works by combining SAM with
Structure from Motion (SfM). The video input to the system is first
reconstructed into 3D geometry using SfM. A frame of video is then segmented
using SAM. Segments identified by SAM are then projected onto the the
reconstructed 3D geometry. In subsequent video frames, the labeled 3D geometry
is reprojected into the new perspective, allowing SAM to be invoked fewer
times. System performance is evaluated, including the contributions of the SAM
and SfM components. Performance is evaluated over three main metrics:
computation time, mask IOU with manual labels, and the number of tracking
losses. Results demonstrate that the system has substantial computation time
improvements over human performance for tracking objects over video frames, but
suffers in performance.