<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 About
This notebook demonstrates stacking machine learning algorithm - folding, which physics use in their analysis.
Step1:  Loading data
Step2:  Training variables
Step3:  Folding strategy - stacking algorithm
It implements the same interface as all classifiers, but with some difference
Step4:  Define folding model
Step5:  Default prediction (predict i_th_ fold by i_th_ classifier)
Step6:  Voting prediction (predict i-fold by all classifiers and take value, which is calculated by vote_function)
Step7:  Comparison of folds
Again use ClassificationReport class to compare different results. For folding classifier this report uses only default prediction.
Report training dataset
Step8:  Signal distribution for each fold
Use mask parameter to plot distribution for the specific fold
Step9:  Background distribution for each fold
Step10:  ROCs (each fold used as test dataset)
Step11:  Report for test dataset
NOTE
<ASSISTANT_TASK:>
Python Code:
%pylab inline
Explanation: About
This notebook demonstrates stacking machine learning algorithm - folding, which physics use in their analysis.
End of explanation
import numpy, pandas
from rep.utils import train_test_split
from sklearn.metrics import roc_auc_score
sig_data = pandas.read_csv('toy_datasets/toyMC_sig_mass.csv', sep='\t')
bck_data = pandas.read_csv('toy_datasets/toyMC_bck_mass.csv', sep='\t')
labels = numpy.array([1] * len(sig_data) + [0] * len(bck_data))
data = pandas.concat([sig_data, bck_data])
train_data, test_data, train_labels, test_labels = train_test_split(data, labels, train_size=0.7)
Explanation: Loading data
End of explanation
variables = ["FlightDistance", "FlightDistanceError", "IP", "VertexChi2", "pt", "p0_pt", "p1_pt", "p2_pt", 'LifeTime', 'dira']
data = data[variables]
Explanation: Training variables
End of explanation
from rep.estimators import SklearnClassifier
from sklearn.ensemble import GradientBoostingClassifier
Explanation: Folding strategy - stacking algorithm
It implements the same interface as all classifiers, but with some difference:
all prediction methods have additional parameter "vote_function" (example folder.predict(X, vote_function=None)), which is used to combine all classifiers' predictions. By default "mean" is used as "vote_function"
End of explanation
from rep.metaml import FoldingClassifier
n_folds = 4
folder = FoldingClassifier(GradientBoostingClassifier(), n_folds=n_folds, features=variables)
folder.fit(train_data, train_labels)
Explanation: Define folding model
End of explanation
folder.predict_proba(train_data)
Explanation: Default prediction (predict i_th_ fold by i_th_ classifier)
End of explanation
# definition of mean function, which combines all predictions
def mean_vote(x):
    return numpy.mean(x, axis=0)
folder.predict_proba(test_data, vote_function=mean_vote)
Explanation: Voting prediction (predict i-fold by all classifiers and take value, which is calculated by vote_function)
End of explanation
from rep.data.storage import LabeledDataStorage
from rep.report import ClassificationReport
# add folds_column to dataset to use mask
train_data["FOLDS"] = folder._get_folds_column(len(train_data))
lds = LabeledDataStorage(train_data, train_labels)
report = ClassificationReport({'folding': folder}, lds)
Explanation: Comparison of folds
Again use ClassificationReport class to compare different results. For folding classifier this report uses only default prediction.
Report training dataset
End of explanation
for fold_num in range(n_folds):
    report.prediction_pdf(mask="FOLDS == %d" % fold_num, labels_dict={1: 'sig fold %d' % fold_num}).plot()
Explanation: Signal distribution for each fold
Use mask parameter to plot distribution for the specific fold
End of explanation
for fold_num in range(n_folds):
    report.prediction_pdf(mask="FOLDS == %d" % fold_num, labels_dict={0: 'bck fold %d' % fold_num}).plot()
Explanation: Background distribution for each fold
End of explanation
for fold_num in range(n_folds):
    report.roc(mask="FOLDS == %d" % fold_num).plot()
Explanation: ROCs (each fold used as test dataset)
End of explanation
lds = LabeledDataStorage(test_data, test_labels)
report = ClassificationReport({'folding': folder}, lds)
report.prediction_pdf().plot(new_plot=True, figsize = (9, 4))
report.roc().plot(xlim=(0.5, 1))
Explanation: Report for test dataset
NOTE: Here vote function is None, so default prediction is used
End of explanation
<END_TASK>