<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Basics Tutorial for Snorkel MeTaL
The purpose of this tutorial is to introduce the basic pipeline, classes, and debugging/analysis tools in Snorkel MeTaL. In this notebook, we'll investigate a single-task problem with synthetic data, with an emphasis on the basic design concepts of MeTaL. Check out the tutorials/ directory to see how this basic pipeline extends to the full multi-task setting, and other topics.
This tutorial consists of four steps, following the basic data programming pipeline as in Snorkel
Step1:  Step 1
Step2:  If you need to divide your data into splits, you can do so with the provided utility function. We split our data 80/10/10 into train/dev/test, stratifying by the labels in Y to ensure a similar class balance in each split.
Step3:  Investigate Label Matrices
MeTaL comes with a number of analysis tools for investigating your label matrix. 
For example, you can view summary statistics for each labeling function.
If you have a dev set with gold labels, you can view the empirical accuracy of your functions, in addition to the core summary statistics. If you do not have a dev set with gold labels, simply do not specify optional argument Y.
Polarity is a list of classes that a labeling function (LF) predicts for one or more examples. Coverage is the fraction of examples that the labeling function predicts over (as opposed to labeling 0 to abstain). Overlaps is the fraction of examples that the labeling function predicts over that at least one other LF predicts over as well. Conflicts is the fraction of examples that the LF predicts as one class but at least one other LF predicts that same example as a different class. Correct, Incorrect, and Empirical Accuracy are all comparisons to the gold standard labels and thus omitted if Y is not specified.
Step4:  If you're interested in more graphical investigative tools, you can take a look at the Visualization tutorial.
Once you're satisfied with your label matrices, it's time to aggregate the labels using the label model and train an end model with the resulting probabilistic labels.
Step 2
Step5:  The only required argument to LabelModel.train_model() is a label matrix. All other keyword arguments are optional. 
For example, if you know the class_balance of your data, you can pass that in as a list or array; alternatively, you may pass in the target labels of your dev set (Y_dev) and have the class balance estimated from that. If neither of these is provided, the classes are assumed to be uniformly distributed.
Any other keyword arguments (e.g., the number of epochs (n_epochs), logging frequency (log_train_every), learning rate (lr), L2 regularization (l2), optimizer type (optimizer), etc.) will be used to update the config dict.
Step6:  You can test the quality of our label model on our dev set as a sanity check, but we'll see if we can do better in Step 3 by using the predictions of the label model to train a discriminative model over a larger feature set than just the outputs of these ten labeling functions.
Step7:  We can see that our trained LabelModel outperforms the baseline of taking the majority vote label by approximately 4 accuracy points.
Step8:  We can see that our trained LabelModel outperforms the baseline of taking the majority vote label by approximately 0.04 in accuracy and 0.03 in F1 on the dev set. However, it has lower recall. Which of these metrics matters most will vary by application.
Concept
Step9:  Analysis tools
At this point, we may want to analyze our LabelModel's predictions so that we better understand what our model is learning.
We can take a look at the confusion matrix to get a sense for class size, error rates, and commonly confused classes.
Step10:  If matplotlib is installed, we can also use some of the visualization tools provided in the contrib/ directory to plot the label distributions.
First, we observe that our problem has a severe class imbalance, which our classifier captures in a general sense.
Second, we can look at how confident our model is in its predictions for class 1 and see that most have probabilities very close to 0.
Step11:  Step 3
Step12:  Once initiated, the network structure is printed so you can confirm that it captured the architecture you want.
To train the model, we pass in our unlabeled training data, our predicted probabilistic (float) labels from the label aggregator, a dev set X and Y if we want to checkpoint the best model seen so far as we train, and any other keyword arguments that you'd like to update in the config dict. For this synthetic problem, the trends in our data are fairly simple, so our model will begin to overfit very quickly. If we do begin to overfit and see our dev score go down, however, the best model seen at any epoch will be restored at the end of training.
Step13:  Step 4
<ASSISTANT_TASK:>
Python Code:
import sys
sys.path.append('../../metal')
import metal
%load_ext autoreload
%autoreload 2
%matplotlib inline
Explanation: Basics Tutorial for Snorkel MeTaL
The purpose of this tutorial is to introduce the basic pipeline, classes, and debugging/analysis tools in Snorkel MeTaL. In this notebook, we'll investigate a single-task problem with synthetic data, with an emphasis on the basic design concepts of MeTaL. Check out the tutorials/ directory to see how this basic pipeline extends to the full multi-task setting, and other topics.
This tutorial consists of four steps, following the basic data programming pipeline as in Snorkel:
1. Load Data: In the weakly supervised setting, we only have access to unlabeled data points X, matrix of noisy labels L, and dev/test labels Y
2. Train Label Model: The purpose of the LabelModel is to estimate the unknown accuracies of the labeling functions, without access to Y, and then use this to denoise and combine them into a set of probabilistic training labels.
3. Train End Model: We can then use these training labels to supervise a discriminative classifier!
4. Evaluate:* We evaluate this model on a held-out test set
Let's get started!
Setup
We first need to make sure that the metal/ directory is on our Python path. If the following cell runs without an error, you're all set. If not, make sure that you've installed snorkel-metal with pip (conda coming soon) or that you've added the repo to your path if you're running from source; for example, running source add_to_path.sh from the repository root.
End of explanation
import pickle
with open("data/basics_tutorial.pkl", 'rb') as f:
    X, Y, L, D = pickle.load(f)
X.shape
Y.shape
L.shape
Explanation: Step 1: Load Data
The first step in a Snorkel MeTaL application is preparing your data.
Concept 1: Required Data Types
In particular, MeTaL makes use of the following types of data (n = # examples, m = # labeling functions):
* X: an [n]-dim iterable of end model inputs (e.g., feature vectors or encoded sentences for an RNN)
* Y: an [n]-dim numpy.ndarray of target labels ($Y \in {1,...,k}^n$)
* L: an [n,m] scipy.sparse matrix of noisy labels ($L \in {0,...,k}^{n \times m}$, with label 0 reserved for abstentions
And optionally (for use with some debugging/analysis tools):
* D: an [n]-dim iterable of human-readable examples (e.g., sentences)
In this tutorial, we use pre-generated synthetic data, where each example is a bag of words drawn from a different gaussian distribution for each class from a 1000-word vocabulary, and our features are simply the 1000-dimensional vector of counts for each word. For some excellent resources on how to write labeling functions, see the Snorkel tutorials repo.
End of explanation
from metal.utils import split_data
Xs, Ys, Ls, Ds = split_data(X, Y, L, D, splits=[0.8, 0.1, 0.1], stratify_by=Y, seed=123)
Explanation: If you need to divide your data into splits, you can do so with the provided utility function. We split our data 80/10/10 into train/dev/test, stratifying by the labels in Y to ensure a similar class balance in each split.
End of explanation
from metal.analysis import lf_summary
lf_summary(Ls[1],Y=Ys[1])
Explanation: Investigate Label Matrices
MeTaL comes with a number of analysis tools for investigating your label matrix. 
For example, you can view summary statistics for each labeling function.
If you have a dev set with gold labels, you can view the empirical accuracy of your functions, in addition to the core summary statistics. If you do not have a dev set with gold labels, simply do not specify optional argument Y.
Polarity is a list of classes that a labeling function (LF) predicts for one or more examples. Coverage is the fraction of examples that the labeling function predicts over (as opposed to labeling 0 to abstain). Overlaps is the fraction of examples that the labeling function predicts over that at least one other LF predicts over as well. Conflicts is the fraction of examples that the LF predicts as one class but at least one other LF predicts that same example as a different class. Correct, Incorrect, and Empirical Accuracy are all comparisons to the gold standard labels and thus omitted if Y is not specified.
End of explanation
from metal.label_model import LabelModel
label_model = LabelModel(k=2, seed=123)
Explanation: If you're interested in more graphical investigative tools, you can take a look at the Visualization tutorial.
Once you're satisfied with your label matrices, it's time to aggregate the labels using the label model and train an end model with the resulting probabilistic labels.
Step 2: Train Label Model
Concept: Config dicts
All Snorkel MeTaL models are driven by configuration dictionaries, or config dicts; that is, all settings for running the given model are stored in a single dictionary, self.config. 
When a model is initialized or the train_model() method is called, any extra keyword arguments passed by the user will be used to update the config dict. Then, throughout the class's methods, whenever a setting needs to be looked up, it is pulled from this dictionary, rather than required as a keyword. This has a number of benefits:
Default values are stored in only one place, the default config dict, rather than in multiple method signatures.
Code maintenance is simplified, since fewer keyword arguments need to be passed up and down the call stack.
Logging is simplified, since the complete settings for a given model can be logged by simply writing the config dict (a python dictionary) to file.
Label Model
When constructing a LabelModel, the only required argument is k, the cardinality of the task. For example, k=2 means a binary classification task where 1 and 2 are the class labels, and 0 is reserved for an abstain.
End of explanation
%%time
label_model.train_model(Ls[0], Y_dev=Ys[1], n_epochs=500, log_train_every=50)
Explanation: The only required argument to LabelModel.train_model() is a label matrix. All other keyword arguments are optional. 
For example, if you know the class_balance of your data, you can pass that in as a list or array; alternatively, you may pass in the target labels of your dev set (Y_dev) and have the class balance estimated from that. If neither of these is provided, the classes are assumed to be uniformly distributed.
Any other keyword arguments (e.g., the number of epochs (n_epochs), logging frequency (log_train_every), learning rate (lr), L2 regularization (l2), optimizer type (optimizer), etc.) will be used to update the config dict.
End of explanation
score = label_model.score((Ls[1], Ys[1]))
Explanation: You can test the quality of our label model on our dev set as a sanity check, but we'll see if we can do better in Step 3 by using the predictions of the label model to train a discriminative model over a larger feature set than just the outputs of these ten labeling functions.
End of explanation
print('Trained Label Model Metrics:')
scores = label_model.score((Ls[1], Ys[1]), metric=['accuracy','precision', 'recall', 'f1'])
from metal.label_model.baselines import MajorityLabelVoter
mv = MajorityLabelVoter(seed=123)
print('Majority Label Voter Metrics:')
scores = mv.score((Ls[1], Ys[1]), metric=['accuracy','precision', 'recall', 'f1'])
Explanation: We can see that our trained LabelModel outperforms the baseline of taking the majority vote label by approximately 4 accuracy points.
End of explanation
# Y_train_ps stands for "Y[labels]_train[split]_p[redicted]s[oft]"
Y_train_ps = label_model.predict_proba(Ls[0])
Y_train_ps
Explanation: We can see that our trained LabelModel outperforms the baseline of taking the majority vote label by approximately 0.04 in accuracy and 0.03 in F1 on the dev set. However, it has lower recall. Which of these metrics matters most will vary by application.
Concept: Classifier Base Class
Both the LabelModel and EndModel in MeTaL are children of the parent Classifier class. The Classifier class defines three important methods related to making predictions:
* predict_proba(): Returns an [n, k+1] numpy array of probabilistic labels (k+1 because of the option to abstain)
* predict(): Returns an [n]-dim numpy array of  int labels in {0,...,k+1}
* score(): Returns the score (default: accuracy) of the predictions with respect to target labels
Following the data programming paradigm, we will use our label model's probabilistic (float) predictions as training labels for a discriminative end model. Because we want probabilistic labels, we use the predict_proba() method here. The predictions are in the form of an [n,k] np.ndarray of floats summing to 1.0 in each row, where Y_ps[i,j] is the predicted probability of the $i$th data point having true label $j$.
End of explanation
from metal.analysis import confusion_matrix
Y_dev_p = label_model.predict(Ls[1])
cm = confusion_matrix(Ys[1], Y_dev_p)
Explanation: Analysis tools
At this point, we may want to analyze our LabelModel's predictions so that we better understand what our model is learning.
We can take a look at the confusion matrix to get a sense for class size, error rates, and commonly confused classes.
End of explanation
try:
    from metal.contrib.visualization.analysis import (
        plot_predictions_histogram, 
        plot_probabilities_histogram,
    )
    plot_predictions_histogram(Y_dev_p, Ys[1], title="Label Distribution")
    Y_dev_ps = label_model.predict_proba(Ls[1])
    plot_probabilities_histogram(Y_dev_ps[:,0], title="Probablistic Label Distribution")
except ModuleNotFoundError:
    print("The tools in contrib/visualization/ require matplotlib. Try `conda/pip install matplotlib`.")
Explanation: If matplotlib is installed, we can also use some of the visualization tools provided in the contrib/ directory to plot the label distributions.
First, we observe that our problem has a severe class imbalance, which our classifier captures in a general sense.
Second, we can look at how confident our model is in its predictions for class 1 and see that most have probabilities very close to 0.
End of explanation
from metal.end_model import EndModel
import torch
if torch.cuda.is_available():
    device = 'cuda'
else:
    device='cpu'
end_model = EndModel([1000,10,2], seed=123, device=device)
Explanation: Step 3: Train End Model
We will now construct and train our EndModel on the predictions from the LabelModel. Here we describe the construction of a single-task end model. For details on the construction of multi-task end models, refer to the multi-task tutorial.
The EndModel consists of three components: an input layer, zero or more middle layers, and a head layer, with each layer consisting of a torch.nn.Module followed by various optional additional operators (e.g., a ReLU nonlinearity, batch normalization, and/or dropout). 
Input layer:
The input module is an IdentityModule by default, which simply accepts arbitrary-length feature vectors from X as torch.Tensors and passes them on to the next module in the network. If, however, you would like provide your data in some other format, you may pass in a custom nn.Module which maps your input type to torch.Tensors (for example, if your data points are images, you may pass in a ResNet as your input module).
Middle layers
The middle layers are layers between the input layer and head layer. Middle modules are nn.Linear by default.
Head layer:
The head layer is the final layer, consisting of an nn.Linear module (plus a softmax operation when making predictions).
When initiated, the EndModel requires a list of output dimensions. 
The first element is the output dimension of the input module (which is equal to the dimensionality of your feature vectors if using the default Identitymodule).
The last element is the cardinality of the classifier.
The elements in the middle define the number and dimensionality of any middle layers.
End of explanation
end_model.train_model((Xs[0], Y_train_ps), valid_data=(Xs[1], Ys[1]), lr=0.01, l2=0.01, batch_size=256, 
                n_epochs=5, checkpoint_metric='accuracy', checkpoint_metric_mode='max')
Explanation: Once initiated, the network structure is printed so you can confirm that it captured the architecture you want.
To train the model, we pass in our unlabeled training data, our predicted probabilistic (float) labels from the label aggregator, a dev set X and Y if we want to checkpoint the best model seen so far as we train, and any other keyword arguments that you'd like to update in the config dict. For this synthetic problem, the trends in our data are fairly simple, so our model will begin to overfit very quickly. If we do begin to overfit and see our dev score go down, however, the best model seen at any epoch will be restored at the end of training.
End of explanation
print("Label Model:")
score = label_model.score((Ls[2], Ys[2]), metric=['accuracy','precision', 'recall', 'f1'])
print()
print("End Model:")
score = end_model.score((Xs[2], Ys[2]), metric=['accuracy','precision', 'recall', 'f1'])
Explanation: Step 4: Evaluate
With a trained EndModel, we can now evaluate performance on our held-out test set, observing a substantial boost in F1 score over using the LabelModel directly. Remember that the label model predicts Y from the labeling functions, L, whereas the end model predicts Y from the raw input, X.
End of explanation
<END_TASK>