<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 What is machine learning?
One definition
Step1:  What are the features?
- TV
Step2:  Linear regression
Pros
Step3:  Splitting X and y into training and testing sets
Step4:  Linear regression in scikit-learn
Step5:  Interpreting model coefficients
Step6:  Making predictions
Step7:  We need an evaluation metric  in order to compare our predictions with the actual values!
Evaluation metric
Root Mean Squared Error (RMSE) is the square root of the mean of the squared errors
Step8:  Classifications on the iris dataset
Framed as a supervised learning problem
Step9:  Logistic regression
For iris dataset, we are predicting a categorical data. Linear Regression is not a good choice, instead we use logistic regression.
Step10:  Evaluation metric
Classification accuracy
Step11:  Alternatives
Step12:  Apply SVM to iris
Step13:  Supervised Learning In-Depth
Step14:  Motivating Random Forests
Step15:  Ensemble the decision tress
Step16:  Unsupervised learning
Introducing K-Means
K Means is an algorithm for unsupervised clustering
Step17:  By eye, it is relatively easy to pick out the four clusters. If you were to perform an exhaustive search for the different segmentations of the data, however, the search space would be exponential in the number of points. Fortunately, there is a well-known Expectation Maximization (EM) procedure which scikit-learn implements, so that KMeans can be solved relatively quickly.
Step18:  Let's use scikit-learn for K-means clustering on Iris dataset
Step19:  Introducing Principal Component Analysis
Principal Component Analysis is a very powerful unsupervised method for dimensionality reduction in data. It's easiest to visualize by looking at a two-dimensional dataset
Step20:  We can see that there is a definite trend in the data. What PCA seeks to do is to find the Principal Axes in the data, and explain how important those axes are in describing the data distribution
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
# read CSV file directly from a URL and save the results
data = pd.read_csv('http://www-bcf.usc.edu/~gareth/ISL/Advertising.csv', index_col=0)
 
# display the first 5 rows
data.head()
data.shape
Explanation: What is machine learning?
One definition: "Machine learning is the semi-automated extraction of knowledge from data"
Knowledge from data: Starts with a question that might be answerable using data
Automated extraction: A computer provides the insight
Semi-automated: Requires many smart decisions by a human
What are the two main categories of machine learning?
Supervised learning: Making predictions using data
Example: Is a given email "spam" or "ham"?
There is an outcome we are trying to predict
Unsupervised learning: Extracting structure from data
Example: Segment grocery store shoppers into clusters that exhibit similar behaviors
There is no "right answer"
Start with supervised learning
Types of supervised learning
Classification: Predict a categorical response
Regression: Predict a continuous response
End of explanation
# conventional way to import seaborn
import seaborn as sns
# allow plots to appear within the notebook
%matplotlib inline
sns.pairplot(data, x_vars=['TV','radio','newspaper'], y_vars='sales', size=7, aspect=0.7, kind='reg')
Explanation: What are the features?
- TV: advertising dollars spent on TV for a single product in a given market (in thousands of dollars)
- Radio: advertising dollars spent on Radio
- Newspaper: advertising dollars spent on Newspaper
What is the response?
- Sales: sales of a single product in a given market (in thousands of items)
What else do we know?
- Because the response variable is continuous, this is a regression problem.
- There are 200 observations (represented by the rows), and each observation is a single market.
End of explanation
feature_cols = ['TV', 'radio', 'newspaper']
# use the list to select a subset of the original DataFrame
X = data[feature_cols]
# equivalent command to do this in one line
X = data[['TV', 'radio', 'newspaper']]
# print the first 5 rows
X.head()
print(type(X))
print(X.shape)
# select a Series from the DataFrame
y = data['sales']
# equivalent command that works if there are no spaces in the column name
y = data.sales
# print the first 5 values
y.head()
# check the type and shape of y
print(type(y))
print(y.shape)
Explanation: Linear regression
Pros: fast, no tuning required, highly interpretable, well-understood
Cons: unlikely to produce the best predictive accuracy (presumes a linear relationship between the features and response)
Form of linear regression
$y = \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_nx_n$
$y$ is the response
$\beta_0$ is the intercept
$\beta_1$ is the coefficient for $x_1$ (the first feature)
$\beta_n$ is the coefficient for $x_n$ (the nth feature)
In this case:
$y = \beta_0 + \beta_1 \times TV + \beta_2 \times Radio + \beta_3 \times Newspaper$
The $\beta$ values are called the model coefficients. These values are "learned" during the model fitting step using the "least squares" criterion. Then, the fitted model can be used to make predictions!
End of explanation
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)
# default split is 75% for training and 25% for testing
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)
Explanation: Splitting X and y into training and testing sets
End of explanation
# import model
from sklearn.linear_model import LinearRegression
# instantiate
linreg = LinearRegression()
# fit the model to the training data (learn the coefficients)
linreg.fit(X_train, y_train)
Explanation: Linear regression in scikit-learn
End of explanation
# print the intercept and coefficients
print(linreg.intercept_)
print(linreg.coef_)
# pair the feature names with the coefficients
list(zip(feature_cols, linreg.coef_))
Explanation: Interpreting model coefficients
End of explanation
y_pred = linreg.predict(X_test)
print(y_pred)
Explanation: Making predictions
End of explanation
from sklearn import metrics
import numpy as np
print(np.sqrt(metrics.mean_squared_error(y_test, y_pred)))
Explanation: We need an evaluation metric  in order to compare our predictions with the actual values!
Evaluation metric
Root Mean Squared Error (RMSE) is the square root of the mean of the squared errors:
$$\sqrt{\frac 1n\sum_{i=1}^n(y_i-\hat{y}_i)^2}$$
End of explanation
# import load_iris function from datasets module
from sklearn.datasets import load_iris
iris = load_iris()
type(iris)
# print the iris data
print(iris.feature_names)
print(len(iris.data))
# print integers representing the species of each observation
print(iris.target)
print(len(iris.target))
# print the encoding scheme for species: 0 = setosa, 1 = versicolor, 2 = virginica
print(iris.target_names)
X = iris.data
# store response vector in "y"
y = iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=4)
print(X_train.shape)
print(X_test.shape)
print(y_train.shape)
print(y_test.shape)
Explanation: Classifications on the iris dataset
Framed as a supervised learning problem: Predict the species of an iris using the measurements
Famous dataset for machine learning because prediction is easy
Learn more about the iris dataset: UCI Machine Learning Repository
End of explanation
from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
y_pred = logreg.predict(X_test)
print(y_pred)
Explanation: Logistic regression
For iris dataset, we are predicting a categorical data. Linear Regression is not a good choice, instead we use logistic regression.
End of explanation
print(metrics.accuracy_score(y_test, y_pred))
Explanation: Evaluation metric
Classification accuracy:
Proportion of correct predictions
Common evaluation metric for classification problems
End of explanation
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import seaborn; 
from scipy import stats
import pylab as pl
seaborn.set()
from sklearn.datasets.samples_generator import make_blobs
X, y = make_blobs(n_samples=50, centers=2,
                  random_state=0, cluster_std=0.60)
xfit = np.linspace(-1, 3.5)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='spring')
# Draw three lines that couple separate the data
for m, b, d in [(1, 0.65, 0.33), (0.5, 1.6, 0.55), (-0.2, 2.9, 0.2)]:
    yfit = m * xfit + b
    plt.plot(xfit, yfit, '-k')
    plt.fill_between(xfit, yfit - d, yfit + d, edgecolor='none', color='#AAAAAA', alpha=0.4)
plt.xlim(-1, 3.5);
Explanation: Alternatives: Support Vector Machine
Support Vector Machines (SVMs) are a powerful supervised learning algorithm used for classification. SVMs draw a boundary between clusters of data.  SVMs attempt to maximize the margin between sets of points.  Many lines can be drawn to separate the points above:
End of explanation
X = iris.data
y = iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=4)
print(X_train.shape)
print(X_test.shape)
print(y_train.shape)
print(y_test.shape)
from sklearn.svm import SVC
clf = SVC(kernel='linear')
clf.fit(X_train, y_train)
y_pred=clf.predict(X_test)
print(y_pred)
print(metrics.accuracy_score(y_test, y_pred))
Explanation: Apply SVM to iris
End of explanation
from sklearn.datasets import load_digits
digits = load_digits()
digits.keys()
X = digits.data
y = digits.target
print(X.shape)
print(y.shape)
# set up the figure
fig = plt.figure(figsize=(6, 6))  # figure size in inches
fig.subplots_adjust(left=0, right=1, bottom=0, top=1, hspace=0.05, wspace=0.05)
# plot the digits: each image is 8x8 pixels
for i in range(64):
    ax = fig.add_subplot(8, 8, i + 1, xticks=[], yticks=[])
    ax.imshow(digits.images[i], cmap=plt.cm.binary, interpolation='nearest')
    
    # label the image with the target value
    ax.text(0, 7, str(digits.target[i]))
Explanation: Supervised Learning In-Depth: Random Forests
Previously we saw a powerful discriminative classifier, Support Vector Machines.
Here we'll take a look at motivating another powerful algorithm. This one is a non-parametric algorithm called Random Forests.
Example: Random Forest for Classifying Digits
Let's start with the hand-written digits data. Let's use that here to test the efficacy of Random Forest classifiers.
End of explanation
from sklearn.cross_validation import train_test_split
from sklearn import metrics
Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, random_state=0)
clf = DecisionTreeClassifier(max_depth=11)
clf.fit(Xtrain, ytrain)
ypred = clf.predict(Xtest)
metrics.accuracy_score(ypred, ytest)
plt.imshow(metrics.confusion_matrix(ypred, ytest),
           interpolation='nearest', cmap=plt.cm.binary)
plt.grid(False)
plt.colorbar()
plt.xlabel("predicted label")
plt.ylabel("true label");
from sklearn.ensemble import RandomForestClassifier
Explanation: Motivating Random Forests: Decision Trees
Random forests are an example of an ensemble learner built on decision trees.
For this reason we'll start by discussing decision trees themselves.
End of explanation
clf = RandomForestClassifier(n_jobs=2, random_state=0)
clf.fit(Xtrain, ytrain)
ypred = clf.predict(Xtest)
metrics.accuracy_score(ypred, ytest)
plt.imshow(metrics.confusion_matrix(ypred, ytest),
           interpolation='nearest', cmap=plt.cm.binary)
plt.grid(False)
plt.colorbar()
plt.xlabel("predicted label")
plt.ylabel("true label");
Explanation: Ensemble the decision tress: Random Forest
End of explanation
from sklearn.datasets.samples_generator import make_blobs
X, y = make_blobs(n_samples=300, centers=4,
                  random_state=0, cluster_std=0.60)
plt.scatter(X[:, 0], X[:, 1], s=50);
Explanation: Unsupervised learning
Introducing K-Means
K Means is an algorithm for unsupervised clustering: that is, finding clusters in data based on the data attributes alone (not the labels).
K Means is a relatively easy-to-understand algorithm.  It searches for cluster centers which are the mean of the points within them, such that every point is closest to the cluster center it is assigned to.
Let's look at how KMeans operates on the simple clusters we looked at previously. To emphasize that this is unsupervised, we'll not plot the colors of the clusters:
End of explanation
from sklearn.cluster import KMeans
est = KMeans(4)  # 4 clusters
est.fit(X)
y_kmeans = est.predict(X)
plt.scatter(X[:, 0], X[:, 1], c=y_kmeans, s=50, cmap='rainbow');
Explanation: By eye, it is relatively easy to pick out the four clusters. If you were to perform an exhaustive search for the different segmentations of the data, however, the search space would be exponential in the number of points. Fortunately, there is a well-known Expectation Maximization (EM) procedure which scikit-learn implements, so that KMeans can be solved relatively quickly.
End of explanation
from sklearn import datasets, cluster
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(2)
# load data
iris = datasets.load_iris()
X_iris = iris.data
y_iris = iris.target
k_means = cluster.KMeans(n_clusters=3)
k_means.fit(X_iris) 
labels = k_means.labels_
# check how many of the samples were correctly labeled
correct_labels = sum(y_iris == labels)
print("Result: %d out of %d samples were correctly labeled." % (correct_labels, y_iris.size))
Explanation: Let's use scikit-learn for K-means clustering on Iris dataset
End of explanation
from __future__ import print_function, division
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
# use seaborn plotting style defaults
np.random.seed(1)
X = np.dot(np.random.random(size=(2, 2)), np.random.normal(size=(2, 200))).T
plt.plot(X[:, 0], X[:, 1], 'o')
plt.axis('equal');
Explanation: Introducing Principal Component Analysis
Principal Component Analysis is a very powerful unsupervised method for dimensionality reduction in data. It's easiest to visualize by looking at a two-dimensional dataset:
End of explanation
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
pca.fit(X)
print(pca.explained_variance_)
print(pca.components_)
plt.plot(X[:, 0], X[:, 1], 'o', alpha=0.5)
for length, vector in zip(pca.explained_variance_, pca.components_):
    v = vector * 3 * np.sqrt(length)
    plt.plot([0, v[0]], [0, v[1]], '-k', lw=3)
plt.axis('equal');
Explanation: We can see that there is a definite trend in the data. What PCA seeks to do is to find the Principal Axes in the data, and explain how important those axes are in describing the data distribution:
End of explanation
<END_TASK>