<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Working with data 2017. Class 8
Contact
Javier Garcia-Bernardo
garcia@uva.nl
1. Clustering
2. Data imputation
3. Dimensionality reduction
Step1:  1. Clustering
Step2:  1a. Clustering with K-means
k-means clustering aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean, serving as a prototype of the cluster. This results in a partitioning of the data space into Voronoi cells.
Other methods
Step3:  1b. Clustering with DBSCAN
The DBSCAN algorithm views clusters as areas of high density separated by areas of low density. Due to this rather generic view, clusters found by DBSCAN can be any shape, as oppos
Step4:  1c. Hierarchical clustering
Keeps aggreagating from a point
Step5:  2. Imputation of missing data (fancy)
<ASSISTANT_TASK:>
Python Code:
##Some code to run at the beginning of the file, to be able to show images in the notebook
##Don't worry about this cell
#Print the plots in this screen
%matplotlib inline 
#Be able to plot images saved in the hard drive
from IPython.display import Image 
#Make the notebook wider
from IPython.core.display import display, HTML 
display(HTML("<style>.container { width:90% !important; }</style>"))
import seaborn as sns
import pylab as plt
import pandas as pd
import numpy as np
import scipy.stats
import statsmodels.formula.api as smf
Explanation: Working with data 2017. Class 8
Contact
Javier Garcia-Bernardo
garcia@uva.nl
1. Clustering
2. Data imputation
3. Dimensionality reduction
End of explanation
#Som elibraries
from sklearn import preprocessing
from sklearn.cluster import DBSCAN, KMeans
#Read teh data, dropna, get sample
df = pd.read_csv("data/big3_position.csv",sep="\t").dropna()
df["Revenue"] = np.log10(df["Revenue"])
df["Assets"] = np.log10(df["Assets"])
df["Employees"] = np.log10(df["Employees"])
df["MarketCap"] = np.log10(df["MarketCap"])
df = df.replace([np.inf,-np.inf],np.nan).dropna().sample(300)
df.head(2)
#Scale variables to give all of them the same weight
X = df.loc[:,["Revenue","Assets","Employees","MarketCap"]]
X = preprocessing.scale(X)
print(X.sum(0))
print(X.std(0))
X
Explanation: 1. Clustering
End of explanation
#Get labels of each row and add a new column with the labels
kmeans = KMeans(n_clusters=2, random_state=0).fit(X)
labels = kmeans.labels_
df["kmeans_labels"] = labels
sns.lmplot(x="MarketCap",y="Assets",hue="kmeans_labels",fit_reg=False,data=df)
Explanation: 1a. Clustering with K-means
k-means clustering aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean, serving as a prototype of the cluster. This results in a partitioning of the data space into Voronoi cells.
Other methods: http://scikit-learn.org/stable/modules/clustering.html
End of explanation
#Get labels of each row and add a new column with the labels
db = DBSCAN(eps=1, min_samples=10).fit(X)
labels = db.labels_
df["dbscan_labels"] = labels
sns.lmplot(x="MarketCap",y="Assets",hue="dbscan_labels",fit_reg=False,data=df)
Image(url="http://scikit-learn.org/stable/_images/sphx_glr_plot_cluster_comparison_0011.png")
Explanation: 1b. Clustering with DBSCAN
The DBSCAN algorithm views clusters as areas of high density separated by areas of low density. Due to this rather generic view, clusters found by DBSCAN can be any shape, as oppos
End of explanation
import scipy
import pylab
import scipy.cluster.hierarchy as sch
# Generate distance matrix based on the difference between rows
D = np.zeros([4,4])
for i in range(4):
    for j in range(4):
        D[i,j] = np.sum(np.abs(X[:,i]-X[:,j])) #Euclidean distance or mutual information are also common
        
print(D)
#Create the linkage and plot
Y = sch.linkage(D, method='centroid') #many methods, single, complete...
Z1 = sch.dendrogram(Y, orientation='right',labels=["Revenue","Assets","Employees","MarketCap"])
Explanation: 1c. Hierarchical clustering
Keeps aggreagating from a point
End of explanation
#Required libraries
!conda install tensorflow -y
!pip install fancyimpute
!pip install pydot_ng
import sklearn.preprocessing
import sklearn
#Read the data again but do not 
df = pd.read_csv("data/big3_position.csv",sep="\t")
df["Revenue"] = np.log10(df["Revenue"])
df["Assets"] = np.log10(df["Assets"])
df["Employees"] = np.log10(df["Employees"])
df["MarketCap"] = np.log10(df["MarketCap"])
le = sklearn.preprocessing.LabelEncoder()
labels = le.fit_transform(df["TypeEnt"])
df["TypeEnt_int"] = labels
print(le.classes_)
df = df.replace([np.inf,-np.inf],np.nan).sample(300)
df.head(2)
X = df.loc[:,["Revenue","Assets","Employees","MarketCap","TypeEnt_int"]].values
X
df.describe()
from fancyimpute import KNN
# X is the complete data matrix
# X_incomplete has the same values as X except a subset have been replace with NaN
# Use 10 nearest rows which have a feature to fill in each row's missing features
X_filled_knn = KNN(k=10).complete(X)
df.loc[:,cols] = X_filled_knn
df.describe()
Explanation: 2. Imputation of missing data (fancy)
End of explanation
<END_TASK>