We present a new unsupervised method for learning general-purpose sentence embeddings.
 Unlike existing methods which rely on local contexts, such as words
 inside the sentence or immediately neighboring sentences, our method selects, for
 each target sentence, influential sentences in the entire document based on a document
 structure. We identify a dependency structure of sentences using metadata
 or text styles. Furthermore, we propose a novel out-of-vocabulary word handling
 technique to model many domain-specific terms, which were mostly discarded by
 existing sentence embedding methods. We validate our model on several tasks
 showing 30% precision improvement in coreference resolution in a technical domain,
 and 7.5% accuracy increase in paraphrase detection compared to baselines. Distributed representations are ever more leveraged to understand text BID20 b; BID16 BID23 . Recently, BID12 proposed a neural network model, SKIP-THOUGHT, that embeds a sentence without supervision by training the network to predict the next sentence for a given sentence. However, unlike human reading with broader context and structure in mind, the existing approaches focus on a small continuous context of neighboring sentences. These approaches work well on less structured text like movie transcripts, but do not work well on structured documents like encylopedic articles and technical reports.To better support semantic understanding of such technical documents, we propose a new unsupervised sentence embedding framework to learn general-purpose sentence representations by leveraging long-distance dependencies between sentences in a document. We observe that understanding a sentence often requires understanding of not only the immediate context but more comprehensive context, including the document title, previous paragraphs or even related articles as shown in Figure 1. For instance, all the sentences in the document can be related to the title of the document (1(a )). The first sentence of each item in a list structure can be influenced by the sentence introducing the list (1(b)) . Moreover , html documents can contain hyperlinks to provide more information about a certain term (1(c)). With the contexts obtained from document structure, we can connect ransomware with payment (1(a)) and the four hashes with Locky (1(b)). In this paper, we presented a novel sentence embedding technique exploiting diverse types of structural contexts and domain-specific OOV words. Our method is unsupervised and applicationindependent, and it can be applied to various NLP applications. We evaluated the method on several NLP tasks including coreference resolution, paraphrase detection and sentence prediction. The results show that our model consistently outperforms the existing approaches confirming that considering the structural context generates better quality sentence representations.
To train a sentence embedding using technical documents, our approach considers document structure to find broader context and handle out-of-vocabulary words. Presents ideas for improving sentence embedding by drawing from more context. Learning sentence representations with sentences dependencies information Extends the idea of forming an unsupervised representation of sentences used in the SkipThough approach by using a broader set of evidence for forming the representation of a sentence