Auxiliary Tasks Enhanced Dual-affinity Learning for Weakly Supervised
  Semantic Segmentation
Lian Xu Mohammed Bennamoun Farid Boussaid Wanli Ouyang Ferdous Sohel Dan Xu
Most existing weakly supervised semantic segmentation (WSSS) methods rely on
Class Activation Mapping (CAM) to extract coarse class-specific localization
maps using image-level labels. Prior works have commonly used an off-line
heuristic thresholding process that combines the CAM maps with off-the-shelf
saliency maps produced by a general pre-trained saliency model to produce more
accurate pseudo-segmentation labels. We propose AuxSegNet+, a weakly supervised
auxiliary learning framework to explore the rich information from these
saliency maps and the significant inter-task correlation between saliency
detection and semantic segmentation. In the proposed AuxSegNet+, saliency
detection and multi-label image classification are used as auxiliary tasks to
improve the primary task of semantic segmentation with only image-level
ground-truth labels. We also propose a cross-task affinity learning mechanism
to learn pixel-level affinities from the saliency and segmentation feature
maps. In particular, we propose a cross-task dual-affinity learning module to
learn both pairwise and unary affinities, which are used to enhance the
task-specific features and predictions by aggregating both query-dependent and
query-independent global context for both saliency detection and semantic
segmentation. The learned cross-task pairwise affinity can also be used to
refine and propagate CAM maps to provide better pseudo labels for both tasks.
Iterative improvement of segmentation performance is enabled by cross-task
affinity learning and pseudo-label updating. Extensive experiments demonstrate
the effectiveness of the proposed approach with new state-of-the-art WSSS
results on the challenging PASCAL VOC and MS COCO benchmarks.