Identifying the relations that connect words is an important step towards understanding human languages and is useful for various NLP tasks such as knowledge base completion and analogical reasoning. Simple unsupervised operators such as vector offset between two-word embeddings have shown to recover some specific relationships between those words, if any. Despite this, how to accurately learn generic relation representations from word representations remains unclear. We model relation representation as a supervised learning problem and learn parametrised operators that map pre-trained word embeddings to relation representations. We propose a method for learning relation representations using a feed-forward neural network that performs relation prediction. Our evaluations on two benchmark datasets reveal that the penultimate layer of the trained neural network-based relational predictor acts as a good representation for the relations between words. Different types of relations exist between words in a language such as Hypernym, Meronym, Synonym, etc. Representing relations between words is important for various NLP tasks such as questions answering BID43 , knowledge base completion BID35 and relational information retrieval BID7 .Two main approaches have been proposed in the literature to represent relations between words. In the first approach, a pair of words is represented by a vector derived from a statistical analysis of a text corpus BID39 . In a text corpus, a relationship between two words X and Y can be expressed using lexical patterns containing X and Y as slot variables. For example, "X is a Y" or "Y such as X" indicate that Y is a Hypernym of X BID34 . The elements of the vector representing the relation between two words correspond to the number of times those two words co-occur with a particular pattern in a corpus. Given such a relation representation, the relational similarity between the relations that exist between the two words in two word-pairs can be measured by the cosine of the angle between the corresponding vectors. We call this the holistic approach because a pair of words is treated as a whole rather than the two constituent words separately when creating a relation representation . Sparsity is a well-known problem for the holistic approach as two words have to co-occur enough in a corpus, or else no relation can be represented for rare or unseen word-pairs.In contrast, the second approach for relation representation directly computes a relation representation from pre-trained word representations (i.e. word embeddings) using some relational operators. Prediction-based word embedding learning methods BID27 BID24 represent the meaning of individual words by dense, low-dimensional real-valued vectors by optimising different language modelling objectives. Although no explicit information is provided to the word embedding learning algorithms regarding the semantic relations that exist among words, prior work BID25 has shown that the learnt word embeddings encode remarkable structural properties pertaining to semantic relations. They showed that the difference (vector offset) between two word vectors (here-onwards denoted by PairDiff) is an accurate method for solving analogical questions in the form "a is to b as c is to ?". For example, king − man + woman results in a vector that is closest to the queen vector. We call this approach compositional because the way in which the relation representation is composed by applying some linear algebraic relational operator on the the semantic representations of the the words that participate in a relation. This interesting property of word embeddings sparked a renewed interest in methods that compose relation representations using word embeddings and besides PairDiff, several other unsupervised methods have been proposed such as 3CosAdd and 3CosMult BID19 .Despite the initial hype, recently, multiple independent works have raised concerns about of word embeddings capturing relational structural properties BID22 BID32 BID23 BID29 . Although PairDiff performs well on the Google analogy dataset, its performance for other relation types has been poor BID3 BID41 BID18 . BID41 tested for the generalisation ability of PairDiff using different relation types and found that semantic relations are captured less accurately compared to syntactic relations. Likewise, BID18 showed that word embeddings are unable to detect paradigmatic relations such as Hypernym, Synonym and Antonyms. Methods such as PairDiff are biased towards attributional similarities between individual words than relational similarities and fails in the presence of nearest neighbours. We further discuss various limitations of the existing unsupervised relation composition methods in Section 2.2.Considering the above-mentioned limitations of the unsupervised relation composition methods, a natural question that arises is whether it is possible to learn supervised relation composition methods to overcome those limitations. In this paper, we model relation representation as learning a parametrised operator f (a, b; θ) such that we can accurately represent the relation between two given words a and b from their word representations a and b, without modifying the input word embeddings. For this purpose, we propose a Multi-class Neural Network Penultimate Layer (MnnPl), a simple and effective parametrised operator for computing relation representations from word representations. Specifically, we train a nonlinear multilayer feed-forward neural network using a labelled dataset consisting of word-pairs for different relation types, where the task is to predict the relation between two input words represented by their pre-trained word embeddings. We find that the penultimate layer of the trained neural network provides an accurate relation representation that generalises beyond the relations in the training dataset. We emphasise that our focus here is not to classify a given pair to a relation in a pre-defined set (relation classification), but rather to obtain a good representation for the relation between the two words in the pair. Our experimental results show that MnnPl significantly outperforms unsupervised relational operators including PairDiff in two standard benchmark datasets, and generalises well to unseen out-of-domain relations. We considered the problem of learning relation embeddings from word embeddings using parametrised operators that can be learnt from relation-labelled word-pairs. We experimentally showed that the penultimate layer of a feed-forward neural network trained for classifying relation types (MnnPl) can accurately represent relations between two given words. In particular, some of the disfluencies of the popular PairDiff operator can be avoided by using MnnPl, which works consistently well for both lexicographic and encyclopaedic relations. The relation representations learnt by MnnPl generalise well to previously unseen (out-of-domain) relations as well, even though the number of training instances is typically small for this purpose.Our analysis highlighted some important limitations in the evaluation protocol used in prior work for relation composition operators. Our work questions the belief that unsupervised operators such as vector offset can discover rich relational structures in the word embedding space. More importantly we show that simple supervised relational composition operators can accurately recover the relational regularities hidden inside word embedding spaces. We hope our work will inspire the NLP community to explore more sophisticated supervised operators to extract useful information from word embeddings in the future.Recently, BID31 show that accessing lexical relations such as hypernym relying only on distributional word embeddings that are trained considering 2-ways cooccurrences between words is insufficient. They illustrate the advantages of using the holistic (pattern-based) to detect such relations. Indeed, it is expected that the holistic and the compositional approaches for representing relations have complementary properties since the holistic uses lexical contexts in which the two words of interest co-occur, while the compositional uses only their embeddings BID33 . Interesting future work includes unifying the two approaches for relation representations.
Identifying the relations that connect words is important for various NLP tasks. We model relation representation as a supervised learning problem and learn parametrised operators that map pre-trained word embeddings to relation representations. This paper presents a novel method for representing lexical relations as vectors using just pre-trained word embeddings and a novel loss function operating over pairs of word pairs. A novel solution to the relation compositon problem when you already have pre trained word/entity embeddings and are interested only in learning to compose relation representations.