<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Assignment 2
Introduction
In this assignment we will replicate a gene expression data analysis experiment. We will use both unsupervised clustering, and a supervised approach using the Support Vector Machine classifier.
The data is highly dimensional, in other words there are many more features than samples/observations ($p \gg N$). This is typical of gene expression data and of some other medical data problems that you might encounter, such as proteomic data or other biomedical data. When the number of features/dimensions is much bigger than the number of samples/observations, this is a high-dimensional problem.
The dataset was described and analysed in the following publication
Step1:  Data Exploration
Now that the data have been loaded in X_train, X_test, y_train, and y_test, we can take a look a closer look at our data. Note
Step2:  To see a preview of the data, we can use the head and tail functions
Step3:  Let's see how the classes are distributed. First let's look at the number of unique values, which should equal 14, as we know we have 14 different cancer types
Step4:  We can see how the cancer types are distrubuted using the itemfreq function of the SciPy stats package
Step5:  Using the cancer_names_longform list we declared above, we can print tumour frequencies nicely
Step6:  You can take a quick look at some statistics values for each gene using the useful describe function (we use transpose to perform the analysis on a gene-by-gene basis). For example you may want to look at mean expression levels for each gene to see if they are over-expressed or under-expressed
Step7:  Summary
Now that we have read the data in a form which we can easily use, we move on to the deliverables that must be completed for Assignment 2.
Deliverables for Assignment 2
Clustering
Task
Step8:  Classification
Task
<ASSISTANT_TASK:>
Python Code:
import urllib2
import csv
import pandas as pd
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
%matplotlib inline
url_X_train = 'http://statweb.stanford.edu/~tibs/ElemStatLearn/datasets/14cancer.xtrain'
url_y_train = 'http://statweb.stanford.edu/~tibs/ElemStatLearn/datasets/14cancer.ytrain'
url_X_test = 'http://statweb.stanford.edu/~tibs/ElemStatLearn/datasets/14cancer.xtest'
url_y_test = 'http://statweb.stanford.edu/~tibs/ElemStatLearn/datasets/14cancer.ytest'
# We know there are 144 tumours in the training set and 54 is the test set, so let's make some column names:
column_names_train = ["Tumour_Sample_" + str(_) for _ in np.arange(144)+1]
column_names_test = ["Tumour_Sample_" + str(_) for _ in np.arange(54)+1]
# We will use Pandas to read and properly format the text-based data.
# The delimiter is a regular expression to look for zero or more repetitions of whitespace (\s).
X_train = pd.read_csv(url_X_train, delimiter='\s*', engine='python', names=column_names_train)
X_test = pd.read_csv(url_X_test, delimiter='\s*', engine='python', names=column_names_test)
# Get the labels and store as a list. There are 14 different cancers in the dataset.
y_train = urllib2.urlopen(url_y_train).read().strip().split()
y_test = urllib2.urlopen(url_y_test).read().strip().split()
# There are 14 different types of cancer, numbered 1 to 14, in the vectors y_test and y_train above. 
# For visualising, you may find the names of the cancer types useful:
cancer_names_longform = ["Breast adenocarcinoma", "Prostate adenocarcinoma", 
                         "Lung adenocarcinoma", "Collerectal adenocarcinoma", 
                         "Lymphoma", "Bladder transitional cell carcinoma", 
                         "Melanoma", "Uterine adenocarcinoma", "Leukemia", 
                         "Renal cell carcinoma", "Pancreatic adenocarcinoma", 
                         "Ovarian adenocarcinoma", "Pleural mesothelioma", 
                         "Central nervous system"]
cancer_names_shortform = ["breast", "prostate", "lung", "collerectal", 
                          "lymphoma", "bladder", "melanoma", 
                          "uterus", "leukemia", "renal", "pancreas", 
                          "ovary", "meso", "cns"]
# For testing you may want a merged training and test set.
# To save memory, these are commented out for now.
# X = pd.concat([X_train, X_test])
# y = y_train + y_test
Explanation: Assignment 2
Introduction
In this assignment we will replicate a gene expression data analysis experiment. We will use both unsupervised clustering, and a supervised approach using the Support Vector Machine classifier.
The data is highly dimensional, in other words there are many more features than samples/observations ($p \gg N$). This is typical of gene expression data and of some other medical data problems that you might encounter, such as proteomic data or other biomedical data. When the number of features/dimensions is much bigger than the number of samples/observations, this is a high-dimensional problem.
The dataset was described and analysed in the following publication:
S. Ramaswamy, P.  Tamayo,  R. Rifkin, S. Mukherjee, C.H. Yeang, M. Angelo, C. Ladd, M. Reich, E. Latulippe, J.P. Mesirov, T. Poggio, W. Gerald, M. Loda, E.S. Lander and T.R. Golub. Multiclass cancer diagnosis using tumor gene expression signatures. PNAS, Proceedings of the National Academy of Sciences. 2001 Dec 18; 98(26): 15149–15154.
The full text is available via PubMed:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC64998/pdf/pq2601015149.pdf
Deliverable
The deliverable of this assignment is to replicate the gene expression analysis performed by Ramaswamy et al. in the paper cited above.
Get the Data
Let's first get the data, which has been made available by the authors of the Elements of Statistical Learning (Hastie, Tibshirani and Friedman, 2nd ed., 2009, Springer Verlag). 
In section 18.3, pp. 654–661 of this book, the authors re-analysed the dataset used by Ramaswamy et al. above and have made the formatted gene expression data available via the book's companion website.
The dataset comprises $p=16,063$ gene expressions for $N=144$ tumour samples in the training set and $N=54$ tumour samples in the test set. The data describe 14 different types of cancer. Regarding this dataset, we can safely say that $p \gg N$.
We will now retrieve the data from the Elements of Statistical Learning's website using pandas and urllib2:
End of explanation
X_train.shape
Explanation: Data Exploration
Now that the data have been loaded in X_train, X_test, y_train, and y_test, we can take a look a closer look at our data. Note: It is convention to use large X for data matrices, and small y for target vectors.
As can be seen, in our training set we have $p=16,063$ genes/features and $N=144$ tumours/samples:
End of explanation
X_train.head()
X_test.tail()
Explanation: To see a preview of the data, we can use the head and tail functions:
End of explanation
len(np.unique(y_train))
Explanation: Let's see how the classes are distributed. First let's look at the number of unique values, which should equal 14, as we know we have 14 different cancer types:
End of explanation
stats.itemfreq(y_train)
Explanation: We can see how the cancer types are distrubuted using the itemfreq function of the SciPy stats package:
End of explanation
for freq in stats.itemfreq(y_train):
    print "%s samples appear %s times (shortform: %s)." % (cancer_names_longform[int(freq[0])-1], 
                                                           freq[1], 
                                                           cancer_names_shortform[int(freq[0])-1])
Explanation: Using the cancer_names_longform list we declared above, we can print tumour frequencies nicely:
End of explanation
# Note: The transpose() function here does not permanently transpose the data stored in X_train.
X_train.transpose().describe()
Explanation: You can take a quick look at some statistics values for each gene using the useful describe function (we use transpose to perform the analysis on a gene-by-gene basis). For example you may want to look at mean expression levels for each gene to see if they are over-expressed or under-expressed:
End of explanation
# Your clustering code. Use as many cells as required, use Markdown cells to document where necessary.
Explanation: Summary
Now that we have read the data in a form which we can easily use, we move on to the deliverables that must be completed for Assignment 2.
Deliverables for Assignment 2
Clustering
Task: Perform hierarchical clustering mimicking the approaches used by Ramaswamy et al. in their paper cited above. Plot a dendogram of your results (SciPy provides dendogram plotting functions, see http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.cluster.hierarchy.dendrogram.html for example) - or visualise your clustering in any other way you deem reasonable.
Both SciKit Learn and SciPy offer hierarchical clustering algorithms, see http://docs.scipy.org/doc/scipy/reference/cluster.hierarchy.html and http://scikit-learn.org/stable/modules/clustering.html.
Notice that not all clustering techniques are useful for all purposes. In the case of this assignment, we know the number of clusters we are searching for - this is a requirement for certain clustering algorithms. Other algorithms may require parameters you might not immediately have available to you.
End of explanation
# Your classification code. Use as many cells as required, use Markdown cells to document where necessary.
Explanation: Classification
Task: Use Support Vector Machines and a One Vs. All (OVA) approach to replicate the results from the Ramaswamy et al. paper. 
SciKit Learn provides an SVM package for Support Vector Machines (see http://scikit-learn.org/stable/modules/svm.html). 
Visualise your results appropriately using plots and tables to describe classification results on the test set.
End of explanation
<END_TASK>