Visual grounding of language is an active research field aiming at enriching text-based representations with visual information. In this paper, we propose a new way to leverage visual knowledge for sentence representations. Our approach transfers the structure of a visual representation space to the textual space by using two complementary sources of information: (1) the cluster information: the implicit knowledge that two sentences associated with the same visual content describe the same underlying reality and (2) the perceptual information contained within the structure of the visual space. We use a joint approach to encourage beneficial interactions during training between textual, perceptual, and cluster information. We demonstrate the quality of the learned representations on semantic relatedness, classification, and cross-modal retrieval tasks. Building linguistic vectors that represent semantics is a long-standing issue in Artificial Intelligence. Distributional Semantic Models BID36 BID41 are well-known recent efforts in this direction, making use of the distributional hypothesis BID16 on text corpora to learn word embeddings. At another granularity level, having high-quality general-purpose sentence representations is crucial for all models that encode sentences into semantic vectors, such as the ones used in machine translation BID0 or question answering BID42 . Moreover, encoding semantics of sentences is paramount because sentences describe relationships between objects and thus convey complex and high-level knowledge better than individual words, which mostly refer to a single concept BID38 .Relying only on text can lead to biased representations and unrealistic predictions (e.g., text-based models could predict that "the sky is green" BID1 ). Besides , it has been shown that human understanding of language is grounded in physical reality and perceptual experience (FincherKiefer, 2001 ). To overcome this limitation, one emerging approach is the visual grounding of language, which consists of leveraging visual information, usually from images, to enhance word representations. Two methods showing substantial improvements have emerged: (1) the sequential technique combines textual and visual representations that were separately learned BID3 BID44 , and (2) the joint method learns a common multimodal representation from multiple sources simultaneously BID29 . In the case of words, the latter has proven to produce representations that perform better on intrinsic and downstream tasks.While there exist numerous approaches to learning sentence representations from text corpora only, and to learning multimodal word embeddings, the problem of the visual grounding of sentences is quite new to the research community. To the best of our knowledge, the only work in the field is BID26 . The authors propose a sequential model: linguistic vectors, learned from a purely textual corpus, are concatenated with grounded vectors, which were independently learned from a captioning dataset. However, the two sources are considered separately, which might prevent beneficial interactions between textual and visual modalities during training.We propose a joint model to learn multimodal sentence representations, based on the assumption that the meaning of a sentence is simultaneously grounded in its textual and visual contexts. In our case, the textual context of a sentence consists of adjacent sentences in a text corpus. Within a distinct dataset, the visual context is learned from a paired video and its associated captions. Indeed, we propose to use videos instead of images because of their temporal aspect, since sentences often describe actions grounded in time. The key challenge is to capture visual information. Usually, to transfer information from the visual space to the textual one, one space is projected onto the other BID26 BID29 . However, as pointed out by BID9 , projections are not sufficient to transfer neighborhood structure between modalities. In our work, we rather propose to exploit the visual space by preserving the overall structure, i.e. conserving the similarities between related elements across spaces. More precisely, we take visual context into account by distinguishing two types of complementary information sources. First, the cluster information , which consists in the implicit knowledge that sentences associated with the same video refer to the same underlying reality. Second, the perceptual information , which is the high-level information extracted from a video using a pre-trained CNN.Regarding these considerations, we formulate three Research Questions (RQ):• RQ1: Is perceptual information useful to improve sentence representations?• RQ2: Are cluster and perceptual information complementary, and does their combination compete with previous models based on projections between visual and textual spaces?• RQ3: Is a joint approach better suited than a sequential one regarding the multimodal acquisition of textual and visual knowledge?Our contribution is threefold: (1) We propose a joint multimodal framework for learning grounded sentence representations; (2) We show that cluster and perceptual information are complementary sources of information; (3) To the best of our knowledge, obtained results achieve state-of-the-art performances on multimodal sentence representations. In this paper, we proposed a joint multimodal model to learn sentence representations and our learned grounded sentence embeddings show state-of-the-art performances. Besides, our main findings are the following: (1) Both perceptual and cluster information are useful to learn sentence representations, in a complementary way. (2) Preserving the structure of the visual space, by modeling textual similarities on visual ones, outperforms a strategy based on projecting one space into the other. (3) A joint approach is more appropriate than a sequential method to learn multimodal representation for sentences. As future work, we would investigate the contribution of the temporal knowledge contained in videos for sentence grounding.
We propose a joint model to incorporate visual knowledge in sentence representations The paper proposes a method to use videos paired with captions to improve sentence embeddings This submission proposes a model for sentence learning sentence representations that are grounded, based on associated video data. Proposes a method for improving text-based sentence embeddings through a joint multimodal framework.