Cycle-Consistency Learning for Captioning and Grounding
Ning Wang Jiajun Deng Mingbo Jia
We present that visual grounding and image captioning, which perform as two
mutually inverse processes, can be bridged together for collaborative training
by careful designs. By consolidating this idea, we introduce CyCo, a
cyclic-consistent learning framework to ameliorate the independent training
pipelines of visual grounding and image captioning. The proposed framework (1)
allows the semi-weakly supervised training of visual grounding; (2) improves
the performance of fully supervised visual grounding; (3) yields a general
captioning model that can describe arbitrary image regions. Extensive
experiments show that our fully supervised grounding model achieves
state-of-the-art performance, and the semi-weakly supervised one also exhibits
competitive performance compared to the fully supervised counterparts. Our
image captioning model has the capability to freely describe image regions and
meanwhile shows impressive performance on prevalent captioning benchmarks.