SSG2: A new modelling paradigm for semantic segmentation
Foivos I. Diakogiannis Suzanne Furby Peter Caccetta Xiaoliang Wu Rodrigo Ibata Ondrej Hlinka John Taylor
State-of-the-art models in semantic segmentation primarily operate on single,
static images, generating corresponding segmentation masks. This one-shot
approach leaves little room for error correction, as the models lack the
capability to integrate multiple observations for enhanced accuracy. Inspired
by work on semantic change detection, we address this limitation by introducing
a methodology that leverages a sequence of observables generated for each
static input image. By adding this "temporal" dimension, we exploit strong
signal correlations between successive observations in the sequence to reduce
error rates. Our framework, dubbed SSG2 (Semantic Segmentation Generation 2),
employs a dual-encoder, single-decoder base network augmented with a sequence
model. The base model learns to predict the set intersection, union, and
difference of labels from dual-input images. Given a fixed target input image
and a set of support images, the sequence model builds the predicted mask of
the target by synthesizing the partial views from each sequence step and
filtering out noise. We evaluate SSG2 across three diverse datasets:
UrbanMonitor, featuring orthoimage tiles from Darwin, Australia with five
spectral bands and 0.2m spatial resolution; ISPRS Potsdam, which includes true
orthophoto images with multiple spectral bands and a 5cm ground sampling
distance; and ISIC2018, a medical dataset focused on skin lesion segmentation,
particularly melanoma. The SSG2 model demonstrates rapid convergence within the
first few tens of epochs and significantly outperforms UNet-like baseline
models with the same number of gradient updates. However, the addition of the
temporal dimension results in an increased memory footprint. While this could
be a limitation, it is offset by the advent of higher-memory GPUs and coding
optimizations.