LocCa: Visual Pretraining with Location-aware Captioners
Bo Wan Michael Tschannen Yongqin Xian Filip Pavetic Ibrahim Alabdulmohsin Xiao Wang André Susano Pinto Andreas Steiner Lucas Beyer Xiaohua Zhai
Image captioning has been shown as an effective pretraining method similar to
contrastive pretraining. However, the incorporation of location-aware
information into visual pretraining remains an area with limited research. In
this paper, we propose a simple visual pretraining method with location-aware
captioners (LocCa). LocCa uses a simple image captioner task interface, to
teach a model to read out rich information, i.e. bounding box coordinates, and
captions, conditioned on the image pixel input. Thanks to the multitask
capabilities of an encoder-decoder architecture, we show that an image
captioner can easily handle multiple tasks during pretraining. Our experiments
demonstrate that LocCa outperforms standard captioners significantly on
localization downstream tasks while maintaining comparable performance on
holistic tasks.