We present a new technique for learning visual-semantic embeddings for cross-modal retrieval.   Inspired by the use of hard negatives in structured prediction, and ranking loss functions used in retrieval, we introduce a simple change to common loss functions used to learn multi-modal embeddings.   That, combined with fine-tuning and the use of augmented data, yields significant gains in retrieval performance.   We showcase our approach, dubbed VSE++, on the MS-COCO and Flickr30K datasets, using ablation studies and comparisons with existing methods.   On MS-COCO our approach outperforms state-of-the-art methods by 8.8% in caption retrieval, and 11.3% in image retrieval (based on R@1). Joint embeddings enable a wide range of tasks in image, video and language understanding. Examples include shape-image embeddings BID18 ) for shape inference, bilingual word embeddings BID33 ), human pose-image embeddings for 3D pose inference BID17 ), fine-grained recognition BID22 ), zero-shot learning BID7 ), and modality conversion via synthesis BID23 a) ). Such embeddings entail mappings from two (or more) domains into a common vector space in which semantically associated inputs (e.g., text and images) are mapped to similar locations. The embedding space thus represents the underlying structure of the domains, where locations and often direction are semantically meaningful.In this paper we focus on learning visual-semantic embeddings, central to tasks such as imagecaption retrieval and generation BID13 ; BID11 , and visual questionanswering BID20 . One approach to visual question-answering, for example, is to first describe an image by a set of captions, and then to find the nearest caption in response to a question BID0 ; BID32 ). In the case of image synthesis from text, one approach is to invert the mapping from a joint visual-semantic embedding to the image space BID23 a) ).Here we focus on visual-semantic embeddings for the generic task of cross-modal retrieval; i.e. the retrieval of images given captions, or of captions from a query image. As is common in information retrieval, we measure performance by R@K, i.e., recall at K -the fraction of queries for which the correct item is retrieved in the closest K points to the query in the embedding space (K is usually a small integer, often 1). More generally, retrieval is a natural way to assess the quality of joint embeddings for image and language data for use in subsequent tasks BID9 ).To this end, the problem is one of ranking, for which the correct target(s) should be closer to the query than other items in the corpus, not unlike learning to rank problems (e.g., BID16 ), and max-margin structured prediction BID2 ; . The formulation and model architecture in this paper are most closely related to those of BID13 , learned with a triplet ranking loss. In contrast to that work, we advocate a novel loss, the use of augmented data, and fine-tuning, that together produce a significant increase in caption retrieval performance over the baseline ranking loss on well-known benchmark datasets. We outperform the best reported result on MS-COCO by almost 9%. We also demonstrate that the benefit from a more powerful image encoder, and fine-tuning the image encoder, is amplified with the use of our stronger loss function. To ensure reproducibility , our code will be made publicly available. We refer to our model as VSE++.Finally, we note that our formulation complements other recent articles that propose new model architectures or similarity functions for this problem. BID28 propose an embedding network to fully replace the similarity function used for the ranking loss. An attention mechanism on both image and caption is used by BID21 , where the authors sequentially and selectively focus on a subset of words and image regions to compute the similarity. In BID10 , the authors use a multi-modal context-modulated attention mechanism to compute the similarity between an image and a caption. Our proposed loss function and triplet sampling could be extended and applied to other such approaches. This paper focused on learning visual-semantic embeddings for cross-modal, image-caption retrieval. Inspired by structured prediction, we proposed a new loss based on violations incurred by relatively hard negatives compared to current methods that used expected errors BID13 BID27 ). We performed experiments on the MS-COCO and Flickr30K datasets and showed that our proposed loss significntly improves performance on these datasets. We observed that the improved loss can better guide a more powerful image encoder, ResNet152, and also guide better when fine-tuning an image encoder. With all modifications, our VSE++ model achieves state-of-the-art performance on the MS-COCO dataset, and is slightly below the best recent model on the Flickr30K dataset. Our proposed loss function can be used to train more sophisticated models that have been using a similar ranking loss for training.
A new loss based on relatively hard negatives that achieves state-of-the-art performance in image-caption retrieval. Learning joint embedding of sentences and images using triplet loss that is applied to hardest negatives instead of averaging over all triplets