Natural language understanding research has recently shifted towards complex Machine Learning and Deep Learning algorithms. Such models often outperform their simpler counterparts significantly. However, their performance relies on the availability of large amounts of labeled data, which are rarely available. To tackle this problem, we propose a methodology for extending training datasets to arbitrarily big sizes and training complex, data-hungry models using weak supervision. We apply this methodology on biomedical relation extraction, a task where training datasets are excessively time-consuming and expensive to create, yet has a major impact on downstream applications such as drug discovery. We demonstrate in two small-scale controlled experiments that our method consistently enhances the performance of an LSTM network, with performance improvements comparable to hand-labeled training data. Finally, we discuss the optimal setting for applying weak supervision using this methodology. The amount of scientific papers in the biomedical field is ever increasing. Published papers contain important information, however, encoded in unstructured text, making it difficult for researchers to locate it. Extracting this information in a structured format and storing it within a knowledge base can have a remarkable impact on a variety of important tasks, ranging from drug design to detecting adverse drug effects. During the past decade, there have been efforts towards automation of Information Extraction BID15 , due to the fact that manual annotation of documents from domain experts is labor-intensive to perform on a large scale BID16 .The broader focus of this work is to help automation of semantic triple extraction from biomedical abstracts. We apply our methodology on two different relations: (a ) Regulations, indicating that a Chemical increases (up-regulates) or decreases (down-regulates) the production of a Protein (CPR) and ( b) Chemically Induced Diseases (CID). Both relations are particularly important for areas such as drug design, safety, and discovery, as it will enable researchers to filter out or select chemical substances with specific properties, faster BID16 . We have shown that weak supervision is a tool which can be used for enhancing the performance of complex models, such as deep neural networks, while utilizing both unlabeled data and multiple base learners. Additionally, we have shown that the proposed methodology is practically feasible for the task at hand, as we have succeeded on defining a combination of base learners, which model the problem space sufficiently and allow us to take advantage of additional, unlabeled data. This comes under the requirement that the unlabeled data are drawn from the same domain/distribution as our labeled data, so that our base learners can generalize and perform adequately on D U . In practice, our methodology shifts the human effort from hand-labeling examples to feature engineering and construction of diverse learners. More importantly, once a satisfactory set of diverse learners is available, we can use this method to scale the training datasets in arbitrarily high levels while consistently improving the performance over the supervised learning paradigm. Moreover, the same pipeline can be re-used on similar tasks with the only requirement of providing the appropriate datasets. On the contrary, in the typical supervised learning paradigm, we would have to repeatedly hand-label large datasets.Despite demonstrating the usability of our method using a controlled, small-scale dataset, it is crucial to further explore the requirements of constructing a large enough unlabelled dataset and perform the same experiments there. That would likely improve the metalearner performance further (which is currently upper-bounded by the small dataset size) and allow us to draw stronger conclusions on the research questions of Subsection 6.2. Additionally it would allow us to inspect how performance improves with the increase of D U in a different scale of magnitude and if there seems to be a certain performance threshold, which we cannot surpass using weak supervision. Our preliminary experiments demonstrate that collecting an appropriate unlabeled dataset given a labeled one is a challenging task itself, along with the definition of "appropriate", and semi-supervised algorithms should not take the existence of an appropriate unlabeled dataset for granted.Further, it would be very important to conclude on a more appropriate metric than the F1 score for the evaluation of marginal weak labels. Currently, the absence of an appropriate metric prevents us from drawing conclusions directly from the weak labels, without having to introduce an additional step (train the meta-learner). This would also allow us to select the optimal hyperparameters of the Generative Model and could have a significant impact upon the final performance.Other areas for further investigation include experimenting with the meta-learner (eg. using pre-trained word embeddings or other model architectures) and defining a more appropriate selection method for the Base Learners. Last, it would be interesting to examine how this system would behave if the Base Learners abstained from voting on the examples they are less certain about. One could simply delete a percentage of the votes which are closer to the classification boundary, or perform a probability calibration on the output of the Base Learners and set a minimum confidence threshold below which they would abstain voting. This could also provide the Generative Model with a modeling advantage, compared to unweighted methods (such as Majority Voting), as described in an analysis related to the trade-offs of weak supervision BID29 .
We propose and apply a meta-learning methodology based on Weak Supervision, for combining Semi-Supervised and Ensemble Learning on the task of Biomedical Relationship Extraction. A semi-supervised method for relation classification, which trains multiple base learners using a small labeled dataset and applies some of them to annotate unlabeled examples for semi-supervised learning. This paper addresses the problem of generating training data for biological relation extraction, and uses predictions from data labeled by weak classifiers as additional training data for a meta learning algorithm. This paper proposes a combination of semi-supervised learning and ensemble learning for information extraction, with experiments conducted on a biomedical relation extraction task