<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Unsupervised Learning
Many instances of unsupervised learning, such as dimensionality reduction, manifold learning and feature extraction, find a new representation of the input data without any additional input.
The most simple example of this, which can barely be called learning, is rescaling the data to have zero mean and unit variance. This is a helpful preprocessing step for many machine learning models.
Applying such a preprocessing has a very similar interface to the supervised learning algorithms we saw so far.
Let's load the iris dataset and rescale it
Step1:  The iris dataset is not "centered" that is it has non-zero mean and the standard deviation is different for each component
Step2:  To use a preprocessing method, we first import the estimator, here StandardScaler and instantiate it
Step3:  As with the classification and regression algorithms, we call fit to learn the model from the data. As this is an unsupervised model, we only pass X, not y. This simply estimates mean and standard deviation.
Step4:  Now we can rescale our data by applying the transform (not predict) method
Step5:  X_scaled has the same number of samples and features, but the mean was subtracted and all features were scaled to have unit standard deviation
Step6:  Principal Component Analysis
An unsupervised transformation that is somewhat more interesting is Principle Component Analysis (PCA).
It is a technique to reduce the dimensionality of the data, by creating a linear projection.
That is, we find new features to represent the data that are a linear combination of the old data (i.e. we rotate it).
The way PCA finds these new directions is by looking for the directions of maximum variance.
Usually only few components that explain most of the variance in the data are kept. To illustrate how a rotation might look like, we first show it on two dimensional data and keep both principal components.
We create a Gaussian blob that is rotated
Step7:  As always, we instantiate our PCA model. By default all directions are kept.
Step8:  Then we fit the PCA model with our data. As PCA is an unsupervised algorithm, there is no output y.
Step9:  Then we can transform the data, projected on the principal components
Step10:  On the left of the plot you can see the four points that were on the top right before. PCA found fit first component to be along the diagonal, and the second to be perpendicular to it. As PCA finds a rotation, the principal components are always at right angles to each other.
Dimensionality Reduction for Visualization with PCA
Consider the digits dataset. It cannot be visualized in a single 2D plot, as it has 64 features. We are going to extract 2 dimensions to visualize it in, using the example from the sklearn examples here
Step11:  Note that this projection was determined without any information about the
labels (represented by the colors)
Step12:  This is a 2-dimensional dataset embedded in three dimensions, but it is embedded
in such a way that PCA cannot discover the underlying data orientation
Step13:  Manifold learning algorithms, however, available in the sklearn.manifold
submodule, are able to recover the underlying 2-dimensional manifold
Step14:  Exercise
Compare the results of Isomap and  PCA on a 5-class subset of the digits dataset (load_digits(5)).
Bonus
<ASSISTANT_TASK:>
Python Code:
from sklearn.datasets import load_iris
iris = load_iris()
X, y = iris.data, iris.target
print(X.shape)
Explanation: Unsupervised Learning
Many instances of unsupervised learning, such as dimensionality reduction, manifold learning and feature extraction, find a new representation of the input data without any additional input.
The most simple example of this, which can barely be called learning, is rescaling the data to have zero mean and unit variance. This is a helpful preprocessing step for many machine learning models.
Applying such a preprocessing has a very similar interface to the supervised learning algorithms we saw so far.
Let's load the iris dataset and rescale it:
End of explanation
print("mean : %s " % X.mean(axis=0))
print("standard deviation : %s " % X.std(axis=0))
Explanation: The iris dataset is not "centered" that is it has non-zero mean and the standard deviation is different for each component:
End of explanation
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
Explanation: To use a preprocessing method, we first import the estimator, here StandardScaler and instantiate it:
End of explanation
scaler.fit(X)
Explanation: As with the classification and regression algorithms, we call fit to learn the model from the data. As this is an unsupervised model, we only pass X, not y. This simply estimates mean and standard deviation.
End of explanation
X_scaled = scaler.transform(X)
Explanation: Now we can rescale our data by applying the transform (not predict) method:
End of explanation
print(X_scaled.shape)
print("mean : %s " % X_scaled.mean(axis=0))
print("standard deviation : %s " % X_scaled.std(axis=0))
Explanation: X_scaled has the same number of samples and features, but the mean was subtracted and all features were scaled to have unit standard deviation:
End of explanation
rnd = np.random.RandomState(42)
X_blob = np.dot(rnd.normal(size=(100, 2)), rnd.normal(size=(2, 2))) + rnd.normal(size=2)
plt.scatter(X_blob[:, 0], X_blob[:, 1])
plt.xlabel("feature 1")
plt.ylabel("feature 2")
Explanation: Principal Component Analysis
An unsupervised transformation that is somewhat more interesting is Principle Component Analysis (PCA).
It is a technique to reduce the dimensionality of the data, by creating a linear projection.
That is, we find new features to represent the data that are a linear combination of the old data (i.e. we rotate it).
The way PCA finds these new directions is by looking for the directions of maximum variance.
Usually only few components that explain most of the variance in the data are kept. To illustrate how a rotation might look like, we first show it on two dimensional data and keep both principal components.
We create a Gaussian blob that is rotated:
End of explanation
from sklearn.decomposition import PCA
pca = PCA()
Explanation: As always, we instantiate our PCA model. By default all directions are kept.
End of explanation
pca.fit(X_blob)
Explanation: Then we fit the PCA model with our data. As PCA is an unsupervised algorithm, there is no output y.
End of explanation
X_pca = pca.transform(X_blob)
plt.scatter(X_pca[:, 0], X_pca[:, 1])
plt.xlabel("first principal component")
plt.ylabel("second principal component")
Explanation: Then we can transform the data, projected on the principal components:
End of explanation
# Taken from example in scikit-learn examples
# Authors: Fabian Pedregosa <fabian.pedregosa@inria.fr>
#          Olivier Grisel <olivier.grisel@ensta.org>
#          Mathieu Blondel <mathieu@mblondel.org>
#          Gael Varoquaux
# License: BSD 3 clause (C) INRIA 2011
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import offsetbox
from sklearn import (manifold, datasets, decomposition, ensemble, lda,
                     random_projection)
digits = datasets.load_digits(n_class=6)
n_digits = 500
X = digits.data[:n_digits]
y = digits.target[:n_digits]
n_samples, n_features = X.shape
n_neighbors = 30
def plot_embedding(X, title=None):
    x_min, x_max = np.min(X, 0), np.max(X, 0)
    X = (X - x_min) / (x_max - x_min)
    plt.figure()
    ax = plt.subplot(111)
    for i in range(X.shape[0]):
        plt.text(X[i, 0], X[i, 1], str(digits.target[i]),
                 color=plt.cm.Set1(y[i] / 10.),
                 fontdict={'weight': 'bold', 'size': 9})
    if hasattr(offsetbox, 'AnnotationBbox'):
        # only print thumbnails with matplotlib > 1.0
        shown_images = np.array([[1., 1.]])  # just something big
        for i in range(X.shape[0]):
            dist = np.sum((X[i] - shown_images) ** 2, 1)
            if np.min(dist) < 1e5:
                # don't show points that are too close
                # set a high threshold to basically turn this off
                continue
            shown_images = np.r_[shown_images, [X[i]]]
            imagebox = offsetbox.AnnotationBbox(
                offsetbox.OffsetImage(digits.images[i], cmap=plt.cm.gray_r),
                X[i])
            ax.add_artist(imagebox)
    plt.xticks([]), plt.yticks([])
    if title is not None:
        plt.title(title)
n_img_per_row = 10
img = np.zeros((10 * n_img_per_row, 10 * n_img_per_row))
for i in range(n_img_per_row):
    ix = 10 * i + 1
    for j in range(n_img_per_row):
        iy = 10 * j + 1
        img[ix:ix + 8, iy:iy + 8] = X[i * n_img_per_row + j].reshape((8, 8))
plt.imshow(img, cmap=plt.cm.binary)
plt.xticks([])
plt.yticks([])
plt.title('A selection from the 64-dimensional digits dataset')
print("Computing PCA projection")
X_pca = decomposition.TruncatedSVD(n_components=2).fit_transform(X)
plot_embedding(X_pca, "Principal Components projection of the digits")
plt.show()
Explanation: On the left of the plot you can see the four points that were on the top right before. PCA found fit first component to be along the diagonal, and the second to be perpendicular to it. As PCA finds a rotation, the principal components are always at right angles to each other.
Dimensionality Reduction for Visualization with PCA
Consider the digits dataset. It cannot be visualized in a single 2D plot, as it has 64 features. We are going to extract 2 dimensions to visualize it in, using the example from the sklearn examples here
End of explanation
from sklearn.datasets import make_s_curve
X, y = make_s_curve(n_samples=1000)
from mpl_toolkits.mplot3d import Axes3D
ax = plt.axes(projection='3d')
ax.scatter3D(X[:, 0], X[:, 1], X[:, 2], c=y)
ax.view_init(10, -60)
Explanation: Note that this projection was determined without any information about the
labels (represented by the colors): this is the sense in which the learning
is unsupervised.  Nevertheless, we see that the projection gives us insight
into the distribution of the different digits in parameter space.
Manifold Learning
One weakness of PCA is that it cannot detect non-linear features.  A set
of algorithms known as Manifold Learning have been developed to address
this deficiency.  A canonical dataset used in Manifold learning is the
S-curve, which we briefly saw in an earlier section:
End of explanation
X_pca = PCA(n_components=2).fit_transform(X)
plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y)
Explanation: This is a 2-dimensional dataset embedded in three dimensions, but it is embedded
in such a way that PCA cannot discover the underlying data orientation:
End of explanation
from sklearn.manifold import Isomap
iso = Isomap(n_neighbors=15, n_components=2)
X_iso = iso.fit_transform(X)
plt.scatter(X_iso[:, 0], X_iso[:, 1], c=y)
Explanation: Manifold learning algorithms, however, available in the sklearn.manifold
submodule, are able to recover the underlying 2-dimensional manifold:
End of explanation
from sklearn.datasets import load_digits
digits = load_digits(5)
X = digits.data
# ...
Explanation: Exercise
Compare the results of Isomap and  PCA on a 5-class subset of the digits dataset (load_digits(5)).
Bonus: Also compare to TSNE, another popular manifold learning technique.
End of explanation
<END_TASK>