Weakly Supervised Co-training with Swapping Assignments for Semantic
  Segmentation
Xinyu Yang Hossein Rahmani Sue Black Bryan M. Williams
Class activation maps (CAMs) are commonly employed in weakly supervised
semantic segmentation (WSSS) to produce pseudo-labels. Due to incomplete or
excessive class activation, existing studies often resort to offline CAM
refinement, introducing additional stages or proposing offline modules. This
can cause optimization difficulties for single-stage methods and limit
generalizability. In this study, we aim to reduce the observed CAM
inconsistency and error to mitigate reliance on refinement processes. We
propose an end-to-end WSSS model incorporating guided CAMs, wherein our
segmentation model is trained while concurrently optimizing CAMs online. Our
method, Co-training with Swapping Assignments (CoSA), leverages a dual-stream
framework, where one sub-network learns from the swapped assignments generated
by the other. We introduce three techniques: i) soft perplexity-based
regularization to penalize uncertain regions; ii) a threshold-searching
approach to dynamically revise the confidence threshold; and iii) contrastive
separation to address the coexistence problem. CoSA demonstrates exceptional
performance, achieving mIoU of 76.2\% and 51.0\% on VOC and COCO validation
datasets, respectively, surpassing existing baselines by a substantial margin.
Notably, CoSA is the first single-stage approach to outperform all existing
multi-stage methods including those with additional supervision. Code is
avilable at \url{https://github.com/youshyee/CoSA}.