Learning with Difference Attention for Visually Grounded Self-supervised
  Representations
Aishwarya Agarwal Srikrishna Karanam Balaji Vasan Srinivasan
Recent works in self-supervised learning have shown impressive results on
single-object images, but they struggle to perform well on complex multi-object
images as evidenced by their poor visual grounding. To demonstrate this
concretely, we propose visual difference attention (VDA) to compute visual
attention maps in an unsupervised fashion by comparing an image with its
salient-regions-masked-out version. We use VDA to derive attention maps for
state-of-the art SSL methods and show they do not highlight all salient regions
in an image accurately, suggesting their inability to learn strong
representations for downstream tasks like segmentation. Motivated by these
limitations, we cast VDA as a differentiable operation and propose a new
learning objective, Differentiable Difference Attention (DiDA) loss, which
leads to substantial improvements in an SSL model's visually grounding to an
image's salient regions.