<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 pomegranate / sklearn Naive Bayes comparison
authors
Step3:  Lets first define a function which will create a dataset to train on. We want to be able to test a range of datasets, from very small to very large, to see which implementation is faster. We also want a function which will take in the models and evaluate them. Lets define both of those now.
Step4:  Lets look first at single dimension Gaussian datasets. We'll look at how many times faster pomegranate is, which means that values > 1 show pomegranate is faster and < 1 show pomegranate is slower. Lets also look at the accuracy of both algorithms. They should have the same accuracy since they implement the same algorithm.
Step5:  It looks as if pomegranate is approximately the same speed for training small models but that the prediction time can be a lot faster in pomegranate than in sklearn.
Now let's take a look at how speeds change as we increase the number of classes that need to be predicted rather than phrasing all of the comparisons on binary classification.
Step6:  It looks like, again, pomegranate is around the same speed as sklearn for fitting models, but that it is consistently much faster to make predictions.
Step7:  This does show that pomegranate is faster at making predictions but that both are so fast that potentially it doesn't really matter.
While it's good to start off by looking at naive Bayes' models defined on single features, the more common setting is one where you have many features. Let's look take a look at the relative speeds on larger number of examples when there are 5 features rather than a single one.
Step8:  It looks like pomegranate can be around twice as fast at fitting multivariate Gaussian Naive Bayes models than sklearn when there is more than one feature.
Finally lets show an increasing number of dimensions with a fixed set of 10 classes and 50,000 samples per class.
Step9:  Looks like pomegranate is consistently faster than sklearn at fitting the model but conveges to be approximately the same speed at making predictions in the high dimensional setting. Their accuracies remain identical indicating that the two are learning the same model.
Out of Core Training
Lastly, both pomegranate and sklearn allow for out of core training by fitting on chunks of a dataset. pomegranate does this by calculating summary statistics on the dataset which are enough to allow for exact parameter updates to be done. sklearn implements this using the model.partial_fit(X, y) API call, whereas pomegranate uses model.summarize(X, y) followed by model.from_summaries() to update the internal parameters.  
Lets compare how long each method takes to train on 25 batches of increasing sizes and the accuracy of both methods.
<ASSISTANT_TASK:>
Python Code:
%pylab inline
import seaborn, time
seaborn.set_style('whitegrid')
from sklearn.naive_bayes import GaussianNB
from pomegranate import *
Explanation: pomegranate / sklearn Naive Bayes comparison
authors: <br>
Nicholas Farn (nicholasfarn@gmail.com) <br>
Jacob Schreiber (jmschreiber91@gmail.com)
<a href="https://github.com/scikit-learn/scikit-learn">sklearn</a> is a very popular machine learning package for Python which implements a wide variety of classical machine learning algorithms. In this notebook we benchmark the Naive Bayes implementations in pomegranate and compare it to the implementation in sklearn.
End of explanation
def create_dataset(n_samples, n_dim, n_classes):
    Create a random dataset with n_samples in each class.
    
    X = numpy.concatenate([numpy.random.randn(n_samples, n_dim) + i for i in range(n_classes)])
    y = numpy.concatenate([numpy.zeros(n_samples) + i for i in range(n_classes)])
    return X, y
def plot(fit, predict, skl_error, pom_error, sizes, xlabel):
    Plot the results.
    
    idx = numpy.arange(fit.shape[1])
    
    plt.figure(figsize=(14, 4))
    plt.plot(fit.mean(axis=0), c='c', label="Fitting")
    plt.plot(predict.mean(axis=0), c='m', label="Prediction")
    plt.plot([0, fit.shape[1]], [1, 1], c='k', label="Baseline")
    
    plt.fill_between(idx, fit.min(axis=0), fit.max(axis=0), color='c', alpha=0.3)
    plt.fill_between(idx, predict.min(axis=0), predict.max(axis=0), color='m', alpha=0.3)
    
    plt.xticks(idx, sizes, rotation=65, fontsize=14)
    plt.xlabel('{}'.format(xlabel), fontsize=14)
    plt.ylabel('pomegranate is x times faster', fontsize=14)
    plt.legend(fontsize=12, loc=4)
    plt.show()
    
    
    plt.figure(figsize=(14, 4))
    plt.plot(1 - skl_error.mean(axis=0), alpha=0.5, c='c', label="sklearn accuracy")
    plt.plot(1 - pom_error.mean(axis=0), alpha=0.5, c='m', label="pomegranate accuracy")
    
    plt.fill_between(idx, 1-skl_error.min(axis=0), 1-skl_error.max(axis=0), color='c', alpha=0.3)
    plt.fill_between(idx, 1-pom_error.min(axis=0), 1-pom_error.max(axis=0), color='m', alpha=0.3)
    
    plt.xticks(idx, sizes, rotation=65, fontsize=14)
    plt.xlabel('{}'.format(xlabel), fontsize=14)
    plt.ylabel('Accuracy', fontsize=14)
    plt.legend(fontsize=14) 
    plt.show()
Explanation: Lets first define a function which will create a dataset to train on. We want to be able to test a range of datasets, from very small to very large, to see which implementation is faster. We also want a function which will take in the models and evaluate them. Lets define both of those now.
End of explanation
sizes = numpy.around(numpy.exp(numpy.arange(8, 16))).astype('int')
n, m = sizes.shape[0], 20
skl_predict, pom_predict = numpy.zeros((m, n)), numpy.zeros((m, n))
skl_fit, pom_fit = numpy.zeros((m, n)), numpy.zeros((m, n))
skl_error, pom_error = numpy.zeros((m, n)), numpy.zeros((m, n))
for i in range(m):
    for j, size in enumerate(sizes):
        X, y = create_dataset(size, 1, 2)
        # bench fit times
        tic = time.time()
        skl = GaussianNB()
        skl.fit(X, y)
        skl_fit[i, j] = time.time() - tic
        tic = time.time()
        pom = NaiveBayes.from_samples(NormalDistribution, X, y)
        pom_fit[i, j] = time.time() - tic
        # bench predict times
        tic = time.time()
        skl_predictions = skl.predict(X)
        skl_predict[i, j] = time.time() - tic
        tic = time.time()
        pom_predictions = pom.predict(X)
        pom_predict[i, j] = time.time() - tic
        # check number wrong
        skl_e = (y != skl_predictions).mean()
        pom_e = (y != pom_predictions).mean()
        skl_error[i, j] = min(skl_e, 1-skl_e)
        pom_error[i, j] = min(pom_e, 1-pom_e)
fit = skl_fit / pom_fit
predict = skl_predict / pom_predict
plot(fit, predict, skl_error, pom_error, sizes, "samples per component")
Explanation: Lets look first at single dimension Gaussian datasets. We'll look at how many times faster pomegranate is, which means that values > 1 show pomegranate is faster and < 1 show pomegranate is slower. Lets also look at the accuracy of both algorithms. They should have the same accuracy since they implement the same algorithm.
End of explanation
sizes = numpy.arange(2, 21).astype('int')
n, m = sizes.shape[0], 20
skl_predict, pom_predict = numpy.zeros((m, n)), numpy.zeros((m, n))
skl_fit, pom_fit = numpy.zeros((m, n)), numpy.zeros((m, n))
skl_error, pom_error = numpy.zeros((m, n)), numpy.zeros((m, n))
for i in range(m):
    for j, size in enumerate(sizes):
        X, y = create_dataset(50000 // size, 1, size)
        # bench fit times
        tic = time.time()
        skl = GaussianNB()
        skl.fit(X, y)
        skl_fit[i, j] = time.time() - tic
        tic = time.time()
        pom = NaiveBayes.from_samples(NormalDistribution, X, y)
        pom_fit[i, j] = time.time() - tic
        # bench predict times
        tic = time.time()
        skl_predictions = skl.predict(X)
        skl_predict[i, j] = time.time() - tic
        tic = time.time()
        pom_predictions = pom.predict(X)
        pom_predict[i, j] = time.time() - tic
        # check number wrong
        skl_e = (y != skl_predictions).mean()
        pom_e = (y != pom_predictions).mean()
        skl_error[i, j] = min(skl_e, 1-skl_e)
        pom_error[i, j] = min(pom_e, 1-pom_e)
fit = skl_fit / pom_fit
predict = skl_predict / pom_predict
plot(fit, predict, skl_error, pom_error, sizes, "number of classes")
Explanation: It looks as if pomegranate is approximately the same speed for training small models but that the prediction time can be a lot faster in pomegranate than in sklearn.
Now let's take a look at how speeds change as we increase the number of classes that need to be predicted rather than phrasing all of the comparisons on binary classification.
End of explanation
X, y = create_dataset(50000, 1, 2)
skl = GaussianNB()
skl.fit(X, y)
pom = NaiveBayes.from_samples(NormalDistribution, X, y)
%timeit skl.predict(X)
%timeit pom.predict(X)
Explanation: It looks like, again, pomegranate is around the same speed as sklearn for fitting models, but that it is consistently much faster to make predictions.
End of explanation
sizes = numpy.around(numpy.exp(numpy.arange(8, 16))).astype('int')
n, m = sizes.shape[0], 20
skl_predict, pom_predict = numpy.zeros((m, n)), numpy.zeros((m, n))
skl_fit, pom_fit = numpy.zeros((m, n)), numpy.zeros((m, n))
skl_error, pom_error = numpy.zeros((m, n)), numpy.zeros((m, n))
for i in range(m):
    for j, size in enumerate(sizes):
        X, y = create_dataset(size, 5, 2)
        # bench fit times
        tic = time.time()
        skl = GaussianNB()
        skl.fit(X, y)
        skl_fit[i, j] = time.time() - tic
        tic = time.time()
        pom = NaiveBayes.from_samples(NormalDistribution, X, y)
        pom_fit[i, j] = time.time() - tic
        # bench predict times
        tic = time.time()
        skl_predictions = skl.predict(X)
        skl_predict[i, j] = time.time() - tic
        tic = time.time()
        pom_predictions = pom.predict(X)
        pom_predict[i, j] = time.time() - tic
        # check number wrong
        skl_e = (y != skl_predictions).mean()
        pom_e = (y != pom_predictions).mean()
        skl_error[i, j] = min(skl_e, 1-skl_e)
        pom_error[i, j] = min(pom_e, 1-pom_e)
fit = skl_fit / pom_fit
predict = skl_predict / pom_predict
plot(fit, predict, skl_error, pom_error, sizes, "samples per component")
Explanation: This does show that pomegranate is faster at making predictions but that both are so fast that potentially it doesn't really matter.
While it's good to start off by looking at naive Bayes' models defined on single features, the more common setting is one where you have many features. Let's look take a look at the relative speeds on larger number of examples when there are 5 features rather than a single one.
End of explanation
sizes = numpy.arange(5, 101, 5).astype('int')
n, m = sizes.shape[0], 20
skl_predict, pom_predict = numpy.zeros((m, n)), numpy.zeros((m, n))
skl_fit, pom_fit = numpy.zeros((m, n)), numpy.zeros((m, n))
skl_error, pom_error = numpy.zeros((m, n)), numpy.zeros((m, n))
for i in range(m):
    for j, size in enumerate(sizes):
        X, y = create_dataset(50000, size, 2)
        # bench fit times
        tic = time.time()
        skl = GaussianNB()
        skl.fit(X, y)
        skl_fit[i, j] = time.time() - tic
        tic = time.time()
        pom = NaiveBayes.from_samples(NormalDistribution, X, y)
        pom_fit[i, j] = time.time() - tic
        # bench predict times
        tic = time.time()
        skl_predictions = skl.predict(X)
        skl_predict[i, j] = time.time() - tic
        tic = time.time()
        pom_predictions = pom.predict(X)
        pom_predict[i, j] = time.time() - tic
        # check number wrong
        skl_e = (y != skl_predictions).mean()
        pom_e = (y != pom_predictions).mean()
        skl_error[i, j] = min(skl_e, 1-skl_e)
        pom_error[i, j] = min(pom_e, 1-pom_e)
fit = skl_fit / pom_fit
predict = skl_predict / pom_predict
plot(fit, predict, skl_error, pom_error, sizes, "dimensions")
Explanation: It looks like pomegranate can be around twice as fast at fitting multivariate Gaussian Naive Bayes models than sklearn when there is more than one feature.
Finally lets show an increasing number of dimensions with a fixed set of 10 classes and 50,000 samples per class.
End of explanation
sizes = numpy.around( numpy.exp( numpy.arange(8, 16) ) ).astype('int')
n, m = sizes.shape[0], 20
skl_time, pom_time = numpy.zeros((m, n)), numpy.zeros((m, n))
skl_error, pom_error = numpy.zeros((m, n)), numpy.zeros((m, n))
for i in range(m):
    for j, size in enumerate(sizes):
        skl = GaussianNB()
        pom = NaiveBayes([IndependentComponentsDistribution([NormalDistribution(0, 1) for i in range(5)]),
                          IndependentComponentsDistribution([NormalDistribution(0, 1) for i in range(5)])])
        
        for l in range(5):
            X, y = create_dataset(size, 5, 2)
            tic = time.time()
            skl.partial_fit(X, y, classes=[0, 1])
            skl_time[i, j] += time.time() - tic
            tic = time.time()
            pom.summarize( X, y )
            pom_time[i, j] += time.time() - tic
        tic = time.time()
        pom.from_summaries()
        pom_time[i, j] += time.time() - tic
        skl_predictions = skl.predict( X )
        pom_predictions = pom.predict( X )
        skl_error[i, j] = ( y != skl_predictions ).mean()
        pom_error[i, j] = ( y != pom_predictions ).mean()
fit = skl_time / pom_time
idx = numpy.arange(fit.shape[1])
plt.figure( figsize=(14, 4))
plt.plot( fit.mean(axis=0), c='c', label="Fitting")
plt.plot( [0, fit.shape[1]], [1, 1], c='k', label="Baseline" )
plt.fill_between( idx, fit.min(axis=0), fit.max(axis=0), color='c', alpha=0.3 )
plt.xticks(idx, sizes, rotation=65, fontsize=14)
plt.xlabel('{}'.format(xlabel), fontsize=14)
plt.ylabel('pomegranate is x times faster', fontsize=14)
plt.legend(fontsize=12, loc=4)
plt.show()
plt.figure( figsize=(14, 4))
plt.plot( 1 - skl_error.mean(axis=0), alpha=0.5, c='c', label="sklearn accuracy" )
plt.plot( 1 - pom_error.mean(axis=0), alpha=0.5, c='m', label="pomegranate accuracy" )
plt.fill_between( idx, 1-skl_error.min(axis=0), 1-skl_error.max(axis=0), color='c', alpha=0.3 )
plt.fill_between( idx, 1-pom_error.min(axis=0), 1-pom_error.max(axis=0), color='m', alpha=0.3 )
plt.xticks( idx, sizes, rotation=65, fontsize=14)
plt.xlabel('Batch Size', fontsize=14)
plt.ylabel('Accuracy', fontsize=14)
plt.legend(fontsize=14) 
plt.show()
Explanation: Looks like pomegranate is consistently faster than sklearn at fitting the model but conveges to be approximately the same speed at making predictions in the high dimensional setting. Their accuracies remain identical indicating that the two are learning the same model.
Out of Core Training
Lastly, both pomegranate and sklearn allow for out of core training by fitting on chunks of a dataset. pomegranate does this by calculating summary statistics on the dataset which are enough to allow for exact parameter updates to be done. sklearn implements this using the model.partial_fit(X, y) API call, whereas pomegranate uses model.summarize(X, y) followed by model.from_summaries() to update the internal parameters.  
Lets compare how long each method takes to train on 25 batches of increasing sizes and the accuracy of both methods.
End of explanation
<END_TASK>