<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 note
Step1:  You want to divide data into 3 set. 
1. Training set
2. Cross Validation set
3. Test set.  
You shouldn't be doing prediction using training data or Validation data as it does in the exercise.
Step2:  divide original validation data into validation and test set
Step3:  Visualize training data
Step4:  estimate multivariate Gaussian parameters $\mu$ and $\sigma^2$
according to data, X1, and X2 is not independent
Step5:  select threshold $\epsilon$
use training set $X$ to model the multivariate Gaussian
use cross validation set $(Xval, yval)$ to find the best $\epsilon$ by finding the best F-score
<img style="float
Step6:  visualize prediction of Xval using learned $\epsilon$
use CV data to find the best $\epsilon$
use all data (training + validation) to create model
do the prediction on test data
Step7:  high dimension data
<ASSISTANT_TASK:>
Python Code:
%reload_ext autoreload
%autoreload 2
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
sns.set(context="notebook", style="white", palette=sns.color_palette("RdBu"))
import numpy as np
import pandas as pd
import scipy.io as sio
from scipy import stats
import sys
sys.path.append('..')
from helper import anomaly
from sklearn.cross_validation import train_test_split
Explanation: note:
covariance matrix
multivariate_normal
seaborn  bivariate kernel density estimate
End of explanation
mat = sio.loadmat('./data/ex8data1.mat')
mat.keys()
X = mat.get('X')
Explanation: You want to divide data into 3 set. 
1. Training set
2. Cross Validation set
3. Test set.  
You shouldn't be doing prediction using training data or Validation data as it does in the exercise.
End of explanation
Xval, Xtest, yval, ytest = train_test_split(mat.get('Xval'),
                                            mat.get('yval').ravel(),
                                            test_size=0.5)
Explanation: divide original validation data into validation and test set
End of explanation
sns.regplot('Latency', 'Throughput',
           data=pd.DataFrame(X, columns=['Latency', 'Throughput']), 
           fit_reg=False,
           scatter_kws={"s":20,
                        "alpha":0.5})
Explanation: Visualize training data
End of explanation
mu = X.mean(axis=0)
print(mu, '\n')
cov = np.cov(X.T)
print(cov)
# example of creating 2d grid to calculate probability density
np.dstack(np.mgrid[0:3,0:3])
# create multi-var Gaussian model
multi_normal = stats.multivariate_normal(mu, cov)
# create a grid
x, y = np.mgrid[0:30:0.01, 0:30:0.01]
pos = np.dstack((x, y))
fig, ax = plt.subplots()
# plot probability density
ax.contourf(x, y, multi_normal.pdf(pos), cmap='Blues')
# plot original data points
sns.regplot('Latency', 'Throughput',
           data=pd.DataFrame(X, columns=['Latency', 'Throughput']), 
           fit_reg=False,
           ax=ax,
           scatter_kws={"s":10,
                        "alpha":0.4})
Explanation: estimate multivariate Gaussian parameters $\mu$ and $\sigma^2$
according to data, X1, and X2 is not independent
End of explanation
e, fs = anomaly.select_threshold(X, Xval, yval)
print('Best epsilon: {}\nBest F-score on validation data: {}'.format(e, fs))
Explanation: select threshold $\epsilon$
use training set $X$ to model the multivariate Gaussian
use cross validation set $(Xval, yval)$ to find the best $\epsilon$ by finding the best F-score
<img style="float: left;" src="../img/f1_score.png">
End of explanation
multi_normal, y_pred = anomaly.predict(X, Xval, e, Xtest, ytest)
# construct test DataFrame
data = pd.DataFrame(Xtest, columns=['Latency', 'Throughput'])
data['y_pred'] = y_pred
# create a grid for graphing
x, y = np.mgrid[0:30:0.01, 0:30:0.01]
pos = np.dstack((x, y))
fig, ax = plt.subplots()
# plot probability density
ax.contourf(x, y, multi_normal.pdf(pos), cmap='Blues')
# plot original Xval points
sns.regplot('Latency', 'Throughput',
            data=data,
            fit_reg=False,
            ax=ax,
            scatter_kws={"s":10,
                         "alpha":0.4})
# mark the predicted anamoly of CV data. We should have a test set for this...
anamoly_data = data[data['y_pred']==1]
ax.scatter(anamoly_data['Latency'], anamoly_data['Throughput'], marker='x', s=50)
Explanation: visualize prediction of Xval using learned $\epsilon$
use CV data to find the best $\epsilon$
use all data (training + validation) to create model
do the prediction on test data
End of explanation
mat = sio.loadmat('./data/ex8data2.mat')
X = mat.get('X')
Xval, Xtest, yval, ytest = train_test_split(mat.get('Xval'),
                                            mat.get('yval').ravel(),
                                            test_size=0.5)
e, fs = anomaly.select_threshold(X, Xval, yval)
print('Best epsilon: {}\nBest F-score on validation data: {}'.format(e, fs))
multi_normal, y_pred = anomaly.predict(X, Xval, e, Xtest, ytest)
print('find {} anamolies'.format(y_pred.sum()))
Explanation: high dimension data
End of explanation
<END_TASK>