<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Relation extraction using distant supervision
Step1:  Contents
Overview
Set-up
Building a classifier
Featurizers
Experiments
Analysis
Examining the trained models
Discovering new relation instances
Overview
OK, it's time to get (halfway) serious. Let's apply real machine learning to train a classifier on the training data, and see how it performs on the test data. We'll begin with one of the simplest machine learning setups
Step2:  With the following steps, we build up the dataset we'll use for experiments; it unites a corpus and a knowledge base in the way we described in the previous notebook.
Step3:  The following code splits up our data in a way that supports experimentation
Step4:  Building a classifier
Featurizers
Featurizers are functions which define the feature representation for our model. The primary input to a featurizer will be the KBTriple for which we are generating features. But since our features will be derived from corpus examples containing the entities of the KBTriple, we must also pass in a reference to a Corpus. And in order to make it easy to combine different featurizers, we'll also pass in a feature counter to hold the results.
Here's an implementation for a very simple bag-of-words featurizer. It finds all the corpus examples containing the two entities in the KBTriple, breaks the phrase appearing between the two entity mentions into words, and counts the words. Note that it makes no distinction between "forward" and "reverse" examples.
Step5:  Here's how this featurizer works on a single example
Step6:  You can experiment with adding new kinds of features just by implementing additional featurizers, following simple_bag_of_words_featurizer as an example.
Now, in order to apply machine learning algorithms such as those provided by sklearn, we need a way to convert datasets of KBTriples into feature matrices. The following steps achieve that
Step7:  Experiments
Now we need some functions to train models, make predictions, and evaluate the results. We'll start with train_models(). This function takes as arguments a dictionary of data splits, a list of featurizers, the name of the split on which to train (by default, 'train'), and a model factory, which is a function which initializes an sklearn classifier (by default, a logistic regression classifier). It returns a dictionary holding the featurizers, the vectorizer that was used to generate the training matrix, and a dictionary holding the trained models, one per relation.
Step8:  Next comes predict(). This function takes as arguments a dictionary of data splits, the outputs of train_models(), and the name of the split for which to make predictions. It returns two parallel dictionaries
Step9:  Now evaluate_predictions(). This function takes as arguments the parallel dictionaries of predictions and true labels produced by predict(). It prints summary statistics for each relation, including precision, recall, and F<sub>0.5</sub>-score, and it returns the macro-averaged F<sub>0.5</sub>-score.
Step10:  Finally, we introduce rel_ext.experiment(), which basically chains together rel_ext.train_models(), rel_ext.predict(), and rel_ext.evaluate_predictions(). For convenience, this function returns the output of rel_ext.train_models() as its result.
Running rel_ext.experiment() in its default configuration will give us a baseline result for machine-learned models.
Step11:  Considering how vanilla our model is, these results are quite surprisingly good! We see huge gains for every relation over our top_k_middles_classifier from the previous notebook. This strong performance is a powerful testament to the effectiveness of even the simplest forms of machine learning.
But there is still much more we can do. To make further gains, we must not treat the model as a black box. We must open it up and get visibility into what it has learned, and more importantly, where it still falls down.
Analysis
Examining the trained models
One important way to gain understanding of our trained model is to inspect the model weights. What features are strong positive indicators for each relation, and what features are strong negative indicators?
Step12:  By and large, the high-weight features for each relation are pretty intuitive — they are words that are used to express the relation in question. (The counter-intuitive results merit a bit of investigation!)
The low-weight features (that is, features with large negative weights) may be a bit harder to understand. In some cases, however, they can be interpreted as features which indicate some other relation which is anti-correlated with the target relation. (As an example, "directed" is a negative indicator for the author relation.)
Optional exercise
<ASSISTANT_TASK:>
Python Code:
__author__ = "Bill MacCartney and Christopher Potts"
__version__ = "CS224u, Stanford, Spring 2022"
Explanation: Relation extraction using distant supervision: experiments
End of explanation
from collections import Counter
import os
import rel_ext
import utils
# Set all the random seeds for reproducibility. Only the
# system seed is relevant for this notebook.
utils.fix_random_seeds()
rel_ext_data_home = os.path.join('data', 'rel_ext_data')
Explanation: Contents
Overview
Set-up
Building a classifier
Featurizers
Experiments
Analysis
Examining the trained models
Discovering new relation instances
Overview
OK, it's time to get (halfway) serious. Let's apply real machine learning to train a classifier on the training data, and see how it performs on the test data. We'll begin with one of the simplest machine learning setups: a bag-of-words feature representation, and a linear model trained using logistic regression.
Just like we did in the unit on supervised sentiment analysis, we'll leverage the sklearn library, and we'll introduce functions for featurizing instances, training models, making predictions, and evaluating results.
Set-up
See the first notebook in this unit for set-up instructions.
End of explanation
corpus = rel_ext.Corpus(os.path.join(rel_ext_data_home, 'corpus.tsv.gz'))
kb = rel_ext.KB(os.path.join(rel_ext_data_home, 'kb.tsv.gz'))
dataset = rel_ext.Dataset(corpus, kb)
Explanation: With the following steps, we build up the dataset we'll use for experiments; it unites a corpus and a knowledge base in the way we described in the previous notebook.
End of explanation
splits = dataset.build_splits()
splits
Explanation: The following code splits up our data in a way that supports experimentation:
End of explanation
def simple_bag_of_words_featurizer(kbt, corpus, feature_counter):
    for ex in corpus.get_examples_for_entities(kbt.sbj, kbt.obj):
        for word in ex.middle.split(' '):
            feature_counter[word] += 1
    for ex in corpus.get_examples_for_entities(kbt.obj, kbt.sbj):
        for word in ex.middle.split(' '):
            feature_counter[word] += 1
    return feature_counter
Explanation: Building a classifier
Featurizers
Featurizers are functions which define the feature representation for our model. The primary input to a featurizer will be the KBTriple for which we are generating features. But since our features will be derived from corpus examples containing the entities of the KBTriple, we must also pass in a reference to a Corpus. And in order to make it easy to combine different featurizers, we'll also pass in a feature counter to hold the results.
Here's an implementation for a very simple bag-of-words featurizer. It finds all the corpus examples containing the two entities in the KBTriple, breaks the phrase appearing between the two entity mentions into words, and counts the words. Note that it makes no distinction between "forward" and "reverse" examples.
End of explanation
kbt = kb.kb_triples[0]
kbt
corpus.get_examples_for_entities(kbt.sbj, kbt.obj)[0].middle
simple_bag_of_words_featurizer(kb.kb_triples[0], corpus, Counter())
Explanation: Here's how this featurizer works on a single example:
End of explanation
kbts_by_rel, labels_by_rel = dataset.build_dataset()
featurized = dataset.featurize(kbts_by_rel, featurizers=[simple_bag_of_words_featurizer])
Explanation: You can experiment with adding new kinds of features just by implementing additional featurizers, following simple_bag_of_words_featurizer as an example.
Now, in order to apply machine learning algorithms such as those provided by sklearn, we need a way to convert datasets of KBTriples into feature matrices. The following steps achieve that:
End of explanation
train_result = rel_ext.train_models(
    splits,
    featurizers=[simple_bag_of_words_featurizer])
Explanation: Experiments
Now we need some functions to train models, make predictions, and evaluate the results. We'll start with train_models(). This function takes as arguments a dictionary of data splits, a list of featurizers, the name of the split on which to train (by default, 'train'), and a model factory, which is a function which initializes an sklearn classifier (by default, a logistic regression classifier). It returns a dictionary holding the featurizers, the vectorizer that was used to generate the training matrix, and a dictionary holding the trained models, one per relation.
End of explanation
predictions, true_labels = rel_ext.predict(
    splits, train_result, split_name='dev')
Explanation: Next comes predict(). This function takes as arguments a dictionary of data splits, the outputs of train_models(), and the name of the split for which to make predictions. It returns two parallel dictionaries: one holding the predictions (grouped by relation), the other holding the true labels (again, grouped by relation).
End of explanation
rel_ext.evaluate_predictions(predictions, true_labels)
Explanation: Now evaluate_predictions(). This function takes as arguments the parallel dictionaries of predictions and true labels produced by predict(). It prints summary statistics for each relation, including precision, recall, and F<sub>0.5</sub>-score, and it returns the macro-averaged F<sub>0.5</sub>-score.
End of explanation
_ = rel_ext.experiment(
    splits,
    featurizers=[simple_bag_of_words_featurizer])
Explanation: Finally, we introduce rel_ext.experiment(), which basically chains together rel_ext.train_models(), rel_ext.predict(), and rel_ext.evaluate_predictions(). For convenience, this function returns the output of rel_ext.train_models() as its result.
Running rel_ext.experiment() in its default configuration will give us a baseline result for machine-learned models.
End of explanation
rel_ext.examine_model_weights(train_result)
Explanation: Considering how vanilla our model is, these results are quite surprisingly good! We see huge gains for every relation over our top_k_middles_classifier from the previous notebook. This strong performance is a powerful testament to the effectiveness of even the simplest forms of machine learning.
But there is still much more we can do. To make further gains, we must not treat the model as a black box. We must open it up and get visibility into what it has learned, and more importantly, where it still falls down.
Analysis
Examining the trained models
One important way to gain understanding of our trained model is to inspect the model weights. What features are strong positive indicators for each relation, and what features are strong negative indicators?
End of explanation
rel_ext.find_new_relation_instances(
    dataset,
    featurizers=[simple_bag_of_words_featurizer])
Explanation: By and large, the high-weight features for each relation are pretty intuitive — they are words that are used to express the relation in question. (The counter-intuitive results merit a bit of investigation!)
The low-weight features (that is, features with large negative weights) may be a bit harder to understand. In some cases, however, they can be interpreted as features which indicate some other relation which is anti-correlated with the target relation. (As an example, "directed" is a negative indicator for the author relation.)
Optional exercise: Investigate one of the counter-intuitive high-weight features. Find the training examples which caused the feature to be included. Given the training data, does it make sense that this feature is a good predictor for the target relation?
<!--
- SPOILER: Using `penalty='l1'` results in somewhat less intuitive feature weights, and about the same performance.
- SPOILER: Using `penalty='l1', C=0.1` results in much more intuitive feature weights, but much worse performance.
-->
Discovering new relation instances
Another way to gain insight into our trained models is to use them to discover new relation instances that don't currently appear in the KB. In fact, this is the whole point of building a relation extraction system: to extend an existing KB (or build a new one) using knowledge extracted from natural language text at scale. Can the models we've trained do this effectively?
Because the goal is to discover new relation instances which are true but absent from the KB, we can't evaluate this capability automatically. But we can generate candidate KB triples and manually evaluate them for correctness.
To do this, we'll start from corpus examples containing pairs of entities which do not belong to any relation in the KB (earlier, we described these as "negative examples"). We'll then apply our trained models to each pair of entities, and sort the results by probability assigned by the model, in order to find the most likely new instances for each relation.
End of explanation
<END_TASK>