We propose a simple and robust training-free approach for building sentence representations. Inspired by the Gram-Schmidt Process in geometric theory, we build an orthogonal basis of the subspace spanned by a word and its surrounding context in a sentence. We model the semantic meaning of a word in a sentence based on two aspects. One is its relatedness to the word vector subspace already spanned by its contextual words. The other is its novel semantic meaning which shall be introduced as a new basis vector perpendicular to this existing subspace.   Following this motivation, we develop an innovative method based on orthogonal basis to combine pre-trained word embeddings into sentence representation. This approach requires zero training and zero parameters, along with efficient inference performance. We evaluate our approach on 11 downstream NLP tasks. Experimental results show that our model outperforms all existing zero-training alternatives in all the tasks and it is competitive to other approaches relying on either large amounts of labelled data or prolonged training time. The concept of word embeddings has been prevalent in NLP community in recent years, as they can characterize semantic similarity between any pair of words, achieving promising results in a large number of NLP tasks BID14 BID18 BID20 . However, due to the hierarchical nature of human language, it is not sufficient to comprehend text solely based on isolated understanding of each word. This has prompted a recent rise in search for semantically robust embeddings for longer pieces of text, such as sentences and paragraphs.Based on learning paradigms, the existing approaches to sentence embeddings can be categorized into two categories: i) parameterized methods and ii) non-parameterized methods.Parameterized sentence embeddings. These models are parameterized and require training to optimize their parameters. SkipThought BID11 is an encoder-decoder model that predicts adjacent sentences. BID15 proposes an unsupervised model, Sent2Vec, to learn an n-gram feature in a sentence to predict the center word from the surrounding context. Quick thoughts (QT) BID12 replaces the encoder with a classifier to predict context sentences from candidate sequences. BID10 proposesà la carte to learn a linear mapping to reconstruct the center word from its context. BID5 generates the sentence encoder InferSent using Natural Language Inference (NLI) dataset. Universal Sentence Encoder utilizes the transformer BID24 for sentence embeddings. The model is first trained on large scale of unsupervised data from Wikipedia and forums, and then trained on the Stanford Natural Language Inference (SNLI) dataset. BID27 propose the gated recurrent averaging network (GRAN), which is trained on Paraphrase Database (PPDB) and English Wikipedia. BID23 leverages a multi-task learning framework to generate sentence embeddings. BID28 learns the paraphrastic sentence representations as the simple average of updated word embeddings.Non-parameterized sentence embedding. Recent work BID0 shows that, surprisingly, a weighted sum or transformation of word representations can outperform many sophisticated neural network structures in sentence embedding tasks. These methods are parameter-free and require no further training upon pre-trained word vectors. BID0 constructs a sentence embedding called SIF as a sum of pre-trained word embeddings, weighted by reverse document frequency. BID19 concatenates different power mean word embeddings as a sentence vector in p-mean. As these methods do not have a parameterized model, they can be easily adapted to novel text domains with both fast inference speed and high-quality sentence embeddings. In view of this trend, our work aims to further advance the frontier of this group and make its new state-of-the-art.In this paper, we propose a novel sentence embedding algorithm, Geometric Embedding (GEM), based entirely on the geometric structure of word embedding space. Given a d-dim word embedding matrix A ∈ R d×n for a sentence with n words, any linear combination of the sentence's word embeddings lies in the subspace spanned by the n word vectors. We analyze the geometric structure of this subspace in R d . When we consider the words in a sentence one-by-one in order, each word may bring in a novel orthogonal basis to the existing subspace. This new basis can be considered as the new semantic meaning brought in by this word, while the length of projection in this direction can indicate the intensity of this new meaning. It follows that a word with a strong intensity should have a larger influence in the sentence's meaning. Thus, these intensities can be converted into weights to linearly combine all word embeddings to obtain the sentence embedding. In this paper, we theoretically frame the above approach in a QR factorization of the word embedding matrix A. Furthermore, since the meaning and importance of a word largely depends on its close neighborhood, we propose the sliding-window QR factorization method to capture the context of a word and characterize its significance within the context.In the last step, we adapt a similar approach as BID0 to remove top principal vectors before generating the final sentence embedding. This step is to ensure commonly shared background components, e.g. stop words, do not bias sentence similarity comparison. As we build a new orthogonal basis for each sentence, we propose to have disparate background components for each sentence. This motivates us to put forward a sentence-specific principal vector removal method, leading to better empirical results.We evaluate our algorithm on 11 NLP tasks. In all of these tasks, our algorithm outperforms all non-parameterized methods and many parameterized approaches. For example, compared to SIF BID0 , the performance is boosted by 5.5% on STS benchmark dataset, and by 2.5% on SST dataset. Plus, the running time of our model compares favorably with existing models.The rest of this paper is organized as following. In Section 2, we describe our sentence embedding algorithm GEM. We evaluate our model on various tasks in Section 3 and Section 4. Finally, we summarize our work in Section 5. Ablation Study. As shown in in Table 4 , every GEM weight (α n , α s , α u ) and proposed principal components removal methods contribute to the performance. As listed on the left, adding GEM weights improves the score by 8.6% on STS dataset compared with averaging three concatenated word vectors. The sentence-dependent principal component removal (SDR) proposed in GEM improves 0.3% compared to directly removing the top h corpus principal components (SIR). Using GEM weights and SDR together yields an overall improvement of 19.7%. As shown on the right in Table 4 , every weight contributes to the performance of our model. For example, three weights altogether improve the score in SUBJ task by 0.38% compared with only using α n . Sensitivity Study. We evaluate the effect of all four hyper-parameters in our model: the window size m in the contextual window matrix, the number of candidate principal components K, the number of principal components to remove h, and the power of the singular value in coarse sentence embedding, i.e. the power t in f (σ j ) = σ t j in Equation FORMULA9 . We sweep the hyper-parameters and test on STSB dev set, SUBJ, and MPQA. Unspecified parameters are fixed at m = 7, K = 45, h = 17 and t = 3. As shown in Figure 2 , our model is quite robust with respect to hyper-parameters. We proposed a simple non-parameterized method 1 to generate sentence embeddings, based entirely on the geometric structure of the subspace spanned by word embeddings. Our sentence embedding evolves from the new orthogonal basis vector brought in by each word, which represents novel semantic meaning. The evaluation shows that our method not only sets up the new state-of-the-art of non-parameterized models but also performs competitively when compared with models requiring either large amount of training data or prolonged training time. In future work, we plan to consider multi-characters, i.e. subwords, into the model and explore other geometric structures in sentences.
A simple and training-free approach for sentence embeddings with competitive performance compared with sophisticated models requiring either large amount of training data or prolonged training time. Presented a new training-free way of generating sentence embedding with systematic analysis Proposes a new geometry-based method for sentence embedding from word embedding vectors by quantifying the novelty, significance, and corpus-uniqueness of each word This paper explores sentence embedding based on orthogonal decomposition of the spanned space by word embeddings