Representing entities and relations in an embedding space is a well-studied approach for machine learning on relational data. Existing approaches however primarily focus on simple link structure between a finite set of entities, ignoring the variety of data types that are often used in relational databases, such as text, images, and numerical values. In our approach, we propose a multimodal embedding using different neural encoders for this variety of data, and combine with existing models to learn embeddings of the entities. We extend existing datasets to create two novel benchmarks, YAGO-10-plus and MovieLens-100k-plus, that contain additional relations such as textual descriptions and images of the original entities. We demonstrate that our model utilizes the additional information effectively to provide further gains in accuracy. Moreover, we test our learned multimodal embeddings by using them to predict missing multimodal attributes. Knowledge bases (KB) are an essential part of many computational systems with applications in variety of domains, such as search, structured data management, recommendations, question answering, and information retrieval. However, KBs often suffer from incompleteness, noise in their entries, and inefficient inference. Due to these deficiencies, learning the relational knowledge representation has been a focus of active research BID1 BID10 BID21 BID29 BID4 . These approaches represent relational triples, consisting of a subject entity, relation, and an object entity, by estimating fixed, low-dimensional representations for each entity and relation from observations, thus encode the uncertainty and infer missing facts accurately and efficiently. The subject and the object entities come from a fixed, enumerable set of entities that appear in the knowledge base.Knowledge bases in the real world, however, are rich with a variety of different data types. Apart from a fixed set of entities, the relations often not only include numerical attributes (such as ages, dates, financial, and geoinformation), but also textual attributes (such as names, descriptions, and titles/designations) and images (profile photos, flags, posters, etc.) . Although these different types of relations cannot directly be represented as links in a graph over a fixed set of nodes, they can be crucial pieces of evidences for knowledge base completion. For example the textual descriptions and images might provide evidence for a person's age, profession, and designation. Further, this additional information still contains similar limitations as the conventional link data; they are often missing, may be noisy when observed, and for some applications, may need to be predicted in order to address a query. There is thus a crucial need for relational modeling that goes beyond just the link-based, graph view of knowledge-base completion, is able to utilize all the observed information, and represent the uncertainty of multimodal relational evidence.In this paper, we introduce a multimodal embedding approach for modeling knowledge bases that contains a variety of data types, such as textual, images, numerical, and categorical values. Although we propose a general framework that can be used to extend many of the existing relational modeling approaches, here we primary apply our method to the DistMult approach . We extend this approach that learns a vector for each entity and relation by augmenting it with additional neural encoders for different evidence data types. For example, when the object of a triple is an image, we encode it into a fixed-length vector using a CNN, while the textual attributes are encoded using sequential embedding approaches like LSTMs. The scoring module remains identical; given the vector representations of the subject, relation, and object of a triple, this module produces a score indicating the probability that the triple is correct. This unified model allows for flow of the information across the different relation types, enabling more accurate modeling of relational data.We provide an evaluation of our proposed approach on two relational databases. Since we are introducing a novel formulation in the relational setting, we introduce two benchmarks, created by extending the existing YAGO-10 and MovieLens-100k datasets to include additional relations such as textual descriptions, numerical attributes, and images of the original entities. In our evaluation, we demonstrate that our model utilizes the additional information effectively to provide gains in link-prediction accuracy, and present a breakdown of how much each relation benefits from each type of the additional information. We also present results that indicate the learned multimodal embeddings are capable of predicting the object entities for different types of data which is based on the similarity between those entities. Motivated by the need to utilize multiple source of information to achieve more accurate link prediction we presented a novel neural approach to multimodal relational learning. In this paper we introduced a universal link prediction model that uses different types of information to model knowledge bases. We proposed a compositional encoding component to learn unified entity embedding that encode the variety of information available for each entity. In our analysis we show that our model in comparison to a common link predictor, DistMult, can achieve higher accuracy, showing the importance of employing the available variety of information for each entity. Since all the existing datasets are designed for previous methods, they lack mentioned kind of extra information. In result, we introduced two new benchmarks YAGO-10-plus and MovieLens-100k-plus, that are extend version of existing datasets. Further, in our evaluation, we showed that our model effectively utilizes the extra information in order to benefit existing relations. We will release the datasets and the open-source implementation of our models publicly.There are number of avenues for future work. We will investigate the performance of our model in completing link prediction task using different scoring function and more elaborate encoding component and objective function. We are also interested in modeling decoding of multimodal values in the model itself, to be able to query these values directly. Further, we plan to explore efficient query algorithms for embedded knowledge bases, to compete with practical database systems.
Extending relational modeling to support multimodal data using neural encoders. This paper proposes to perform link prediction in Knowledge Bases by supplementing the original entities with multimodal information, and presents a model able to encode all sorts of information when scoring triples. The paper is about incorporating information from different modalities into link prediction approaches