Current state-of-the-art relation extraction methods typically rely on a set of lexical, syntactic, and semantic features, explicitly computed in a pre-processing step. Training feature extraction models requires additional annotated language resources, which severely restricts the applicability and portability of relation extraction to novel languages. Similarly, pre-processing introduces an additional source of error. To address these limitations, we introduce TRE, a Transformer for Relation Extraction, extending the OpenAI Generative Pre-trained Transformer [Radford et al., 2018]. Unlike previous relation extraction models, TRE uses pre-trained deep language representations instead of explicit linguistic features to inform the relation classification and combines it with the self-attentive Transformer architecture to effectively model long-range dependencies between entity mentions. TRE allows us to learn implicit linguistic features solely from plain text corpora by unsupervised pre-training, before fine-tuning the learned language representations on the relation extraction task. TRE obtains a new state-of-the-art result on the TACRED and SemEval 2010 Task 8 datasets, achieving a test F1 of 67.4 and 87.1, respectively. Furthermore, we observe a significant increase in sample efficiency. With only 20% of the training examples, TRE matches the performance of our baselines and our model trained from scratch on 100% of the TACRED dataset. We open-source our trained models, experiments, and source code. Relation extraction aims to identify the relationship between two nominals, typically in the context of a sentence, making it essential to natural language understanding. Consequently, it is a key component of many natural language processing applications, such as information extraction BID2 , knowledge base population BID6 , and question answering BID26 . Table 1 lists exemplary relation mentions.State-of-the-art relation extraction models, traditional feature-based and current neural methods, typically rely on explicit linguistic features: prefix and morphological features BID11 , syntactic features such as part-of-speech tags BID28 , and semantic features like named entity tags and WordNet hypernyms BID25 . Most recently, BID32 combined dependency parse features with graph convolutional neural networks to considerably increase the performance of relation extraction systems. Table 1 : Relation extraction examples, taken from TACRED (1-3) and SemEval 2010 Task 8 (4-6) . TACRED relation types mostly focus on named entities, whereas SemEval contains semantic relations between concepts.However, relying on explicit linguistic features severely restricts the applicability and portability of relation extraction to novel languages. Explicitly computing such features requires large amounts of annotated, language-specific resources for training; many unavailable in non-English languages. Moreover, each feature extraction step introduces an additional source of error, possibly cascading over multiple steps. Deep language representations, on the other hand, have shown to be a very effective form of unsupervised pre-training, yielding contextualized features that capture linguistic properties and benefit downstream natural language understanding tasks, such as semantic role labeling, coreference resolution, and sentiment analysis BID13 . Similarly, fine-tuning pre-trained language representations on a target task has shown to yield state-of-the-art performance on a variety of tasks, such as semantic textual similarity, textual entailment, and question answering BID14 .In addition, classifying complex relations requires a considerable amount of annotated training examples, which are time-consuming and costly to acquire. BID5 showed language model fine-tuning to be a sample efficient method that requires fewer labeled examples.Besides recurrent (RNN) and convolutional neural networks (CNN), the Transformer BID21 is becoming a popular approach to learn deep language representations. Its self-attentive structure allows it to capture long-range dependencies efficiently; demonstrated by the recent success in machine translation BID21 , text generation , and question answering BID14 .In this paper, we propose TRE: a Transformer based Relation Extraction model. Unlike previous methods, TRE uses deep language representations instead of explicit linguistic features to inform the relation classifier. Since language representations are learned by unsupervised language modeling, pre-training TRE only requires a plain text corpus instead of annotated language-specific resources. Fine-tuning TRE, and its representations, directly to the task minimizes explicit feature extraction, reducing the risk of error accumulation. Furthermore , an increased sample efficiency reduces the need for distant supervision methods BID11 BID15 , allowing for simpler model architectures without task-specific modifications.The contributions of our paper are as follows:• We describe TRE, a Transformer based relation extraction model that, unlike previous methods, relies on deep language representations instead of explicit linguistic features.• We are the first to demonstrate the importance of pre-trained language representations in relation extraction, by outperforming state-of-the-art methods on two supervised datasets, TACRED and SemEval 2010 Task 8.• We report detailed ablations, demonstrating that pre-trained language representations prevent overfitting and achieve better generalization in the presence of complex entity mentions. Similarly, we demonstrate a considerable increase in sample efficiency over baseline methods.• We make our trained models, experiments, and source code available to facilitate wider adoption and further research. We proposed TRE, a Transformer based relation extraction method that replaces explicit linguistic features, required by previous methods, with implicit features captured in pretrained language representations. We showed that our model outperformes the state-ofthe-art on two popular relation extraction datasets, TACRED and SemEval 2010 Task 8. We also found that pre-trained language representations drastically improve the sample efficiency of our approach. In our experiments we observed language representations to capture features very informative to the relation extraction task.While our results are strong, important future work is to further investigate the linguistic features that are captured by TRE. One question of interest is the extent of syntactic structure that is captured in language representations, compared to information provided by dependency parsing. Furthermore, our generic architecture enables us to integrate additional contextual information and background knowledge about entities, which could be used to further improve performance.
We propose a Transformer based relation extraction model that uses pre-trained language representations instead of explicit linguistic features. Presents a transformer-based relation extraction model that leverages pre-training on unlabeled text with a language modeling objective. This article describes a novel application of Transformer networks for relation extraction. The paper presents a Transformer based architecture for relaxation extraction, evaluating on two datasets.