<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Working with data 2017. Class 8
Contact
Javier Garcia-Bernardo
garcia@uva.nl
1. Clustering
2. Data imputation
3. Dimensionality reduction
Step1:  3. Dimensionality reduction
Many times we want to combine variables (for linear regression to avoid multicollinearity, to create indexes, etc)
Our data
Step2:  Correlation between variables
Step3:  Revenue, employees and assets are highly correlated.
Let's imagine wwe want to explain the market capitalization in terms of the other variables.
Step4:  3.1 Combining variables
Multiplying/summing variables 
It's easy
It's arbitrary
Step5:  3.2 PCA
Keep all the info
The resulting variables do not actually mean much
Step6:  3.3 Factor analysis
The observations are assumed to be caused by a linear transformation of lower dimensional latent factors and added Gaussian noise. Without loss of generality the factors are distributed according to a Gaussian with zero mean and unit covariance. The noise is also zero mean and has an arbitrary diagonal covariance matrix.
Step7:  Difference between FA and PCA
http
Step8:  Linear regression (to compare)
Step9:  SVR
- Gives balanced weights (the most correlated independent variable (with the dependent) doesn't take all the weight).
- Very good when you have hundreds of variables. You can iteratively drop the worst predictor.
- It allow for more than linear "regression". The default kernel is "rbf", which fits curves. The problem is that interpreting it is hard.
Step10:  Lasso
- Have a penalty
- Discards the variables with low weights.
Step11:  Summary
<ASSISTANT_TASK:>
Python Code:
##Some code to run at the beginning of the file, to be able to show images in the notebook
##Don't worry about this cell
#Print the plots in this screen
%matplotlib inline 
#Be able to plot images saved in the hard drive
from IPython.display import Image 
#Make the notebook wider
from IPython.core.display import display, HTML 
display(HTML("<style>.container { width:90% !important; }</style>"))
import seaborn as sns
import pylab as plt
import pandas as pd
import numpy as np
import scipy.stats
import statsmodels.formula.api as smf
import sklearn
from sklearn.model_selection import train_test_split
Explanation: Working with data 2017. Class 8
Contact
Javier Garcia-Bernardo
garcia@uva.nl
1. Clustering
2. Data imputation
3. Dimensionality reduction
End of explanation
#Read data
df_companies = pd.read_csv("data/big3_position.csv",sep="\t")
df_companies["log_revenue"] = np.log10(df_companies["Revenue"])
df_companies["log_assets"] = np.log10(df_companies["Assets"])
df_companies["log_employees"] = np.log10(df_companies["Employees"])
df_companies["log_marketcap"] = np.log10(df_companies["MarketCap"])
#Keep only industrial companies
df_companies = df_companies.loc[:,["log_revenue","log_assets","log_employees","log_marketcap","Company_name","TypeEnt"]]
df_companies = df_companies.loc[df_companies["TypeEnt"]=="Industrial company"]
#Dropnans
df_companies = df_companies.replace([np.inf,-np.inf],np.nan)
df_companies = df_companies.dropna()
df_companies.head()
Explanation: 3. Dimensionality reduction
Many times we want to combine variables (for linear regression to avoid multicollinearity, to create indexes, etc)
Our data
End of explanation
# Compute the correlation matrix
corr = df_companies.corr()
# Generate a mask for the upper triangle (hide the upper triangle)
mask = np.zeros_like(corr, dtype=np.bool)
mask[np.triu_indices_from(mask)] = True
# Draw the heatmap with the mask and correct aspect ratio
sns.heatmap(corr, mask=mask, square=True,linewidths=.5,cmap="YlOrRd",vmin=0,vmax=1)
plt.show()
Explanation: Correlation between variables
End of explanation
mod = smf.ols(formula='log_marketcap ~ log_revenue + log_employees  + log_assets', data=df_companies)
res = mod.fit()
print(res.summary())
#The residuals are fine
plt.figure(figsize=(4,3))
sns.regplot(res.predict(),df_companies["log_marketcap"] -res.predict())
#Get many models to see hwo coefficient changes
from statsmodels.iolib.summary2 import summary_col
mod1 = smf.ols(formula='log_marketcap ~ log_revenue + log_employees  + log_assets', data=df_companies).fit()
mod2 = smf.ols(formula='log_marketcap ~ log_revenue + log_assets', data=df_companies).fit()
mod3 = smf.ols(formula='log_marketcap ~ log_employees  + log_assets', data=df_companies).fit()
mod4 = smf.ols(formula='log_marketcap ~ log_assets', data=df_companies).fit()
mod5 = smf.ols(formula='log_marketcap ~ log_revenue + log_employees ', data=df_companies).fit()
mod6 = smf.ols(formula='log_marketcap ~ log_revenue ', data=df_companies).fit()
mod7 = smf.ols(formula='log_marketcap ~ log_employees  ', data=df_companies).fit()
output = summary_col([mod1,mod2,mod3,mod4,mod5,mod6,mod7],stars=True)
print(mod1.rsquared_adj,mod2.rsquared_adj,mod3.rsquared_adj,mod4.rsquared_adj,mod5.rsquared_adj,mod6.rsquared_adj,mod7.rsquared_adj)
output
Explanation: Revenue, employees and assets are highly correlated.
Let's imagine wwe want to explain the market capitalization in terms of the other variables.
End of explanation
X = df_companies.loc[:,["log_revenue","log_employees","log_assets"]]
X.head(2)
#Let's scale all the columns to have mean 0 and std 1
from sklearn.preprocessing import scale
X_to_combine = scale(X)
X_to_combine
#In this case we sum them together
X_combined = np.sum(X_to_combine,axis=1)
X_combined
#Add a new column with our combined variable and run regression
df_companies["combined"] = X_combined
print(smf.ols(formula='log_marketcap ~ combined  ', data=df_companies).fit().summary())
Explanation: 3.1 Combining variables
Multiplying/summing variables 
It's easy
It's arbitrary
End of explanation
#Do the fitting
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
new_X = pca.fit_transform(X)
print("Explained variance")
print(pca.explained_variance_ratio_)
print()
print("Weight of components")
print(["log_revenue","log_employees","log_assets"])
print(pca.components_)
print()
new_X
#Create our new variables (2 components, so 2 variables)
df_companies["pca_x1"] = new_X[:,0]
df_companies["pca_x2"] = new_X[:,1]
print(smf.ols(formula='log_marketcap ~ pca_x1 + pca_x2 ', data=df_companies).fit().summary())
print("Before")
sns.lmplot("log_revenue","log_assets",data=df_companies,fit_reg=False)
print("After")
sns.lmplot("pca_x1","pca_x2",data=df_companies,fit_reg=False)
Explanation: 3.2 PCA
Keep all the info
The resulting variables do not actually mean much
End of explanation
from sklearn.decomposition import FactorAnalysis
fa = FactorAnalysis(n_components=2)
new_X = fa.fit_transform(X)
print("Weight of components")
print(["log_revenue","log_employees","log_assets"])
print(fa.components_)
print()
new_X
#New variables
df_companies["fa_x1"] = new_X[:,0]
df_companies["fa_x2"] = new_X[:,1]
print(smf.ols(formula='log_marketcap ~ fa_x1  + fa_x2 ', data=df_companies).fit().summary())
print("After")
sns.lmplot("fa_x1","fa_x2",data=df_companies,fit_reg=False)
Explanation: 3.3 Factor analysis
The observations are assumed to be caused by a linear transformation of lower dimensional latent factors and added Gaussian noise. Without loss of generality the factors are distributed according to a Gaussian with zero mean and unit covariance. The noise is also zero mean and has an arbitrary diagonal covariance matrix.
End of explanation
Image(url="http://www.holehouse.org/mlclass/07_Regularization_files/Image.png")
from sklearn.model_selection import train_test_split
y = df_companies["log_marketcap"]
X = df_companies.loc[:,["log_revenue","log_employees","log_assets"]]
X.head(2)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)
X_train.head()
Explanation: Difference between FA and PCA
http://stats.stackexchange.com/questions/1576/what-are-the-differences-between-factor-analysis-and-principal-component-analysi
Principal component analysis involves extracting linear composites of observed variables.
Factor analysis is based on a formal model predicting observed variables from theoretical latent factors.
3.4 Methods to avoid overfitting (Machine learning with regularization)
SVR: http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html
Lasso regression: http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html
Both have a regularization parameter, that penalizes having many terms. How to choose the best value of this parameter?
- With a train_test split (or cross-validation)
    - http://scikit-learn.org/stable/modules/cross_validation.html
    - http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html
End of explanation
df_train = X_train.copy()
df_train["log_marketcap"] = y_train
df_train.head()
mod = smf.ols(formula='log_marketcap ~ log_revenue + log_employees  + log_assets', data=df_train).fit()
print("log_revenue 	log_employees 	log_assets ")
print(mod.params.values[1:])
Explanation: Linear regression (to compare)
End of explanation
from sklearn.svm import SVR
clf = SVR(C=0.1, epsilon=0.2,kernel="linear")
clf.fit(X_train, y_train) 
print("log_revenue 	log_employees 	log_assets ")
print(clf.coef_)
Explanation: SVR
- Gives balanced weights (the most correlated independent variable (with the dependent) doesn't take all the weight).
- Very good when you have hundreds of variables. You can iteratively drop the worst predictor.
- It allow for more than linear "regression". The default kernel is "rbf", which fits curves. The problem is that interpreting it is hard.
End of explanation
from sklearn import linear_model
reg = linear_model.Lasso(alpha = 0.01)
reg.fit(X_train,y_train)
print("log_revenue 	log_employees 	log_assets ")
print(reg.coef_)
Explanation: Lasso
- Have a penalty
- Discards the variables with low weights.
End of explanation
print(["SVR","Lasso","Linear regression"])
err1,err2,err3 = sklearn.metrics.mean_squared_error(clf.predict(X_test),y_test),sklearn.metrics.mean_squared_error(reg.predict(X_test),y_test),sklearn.metrics.mean_squared_error(mod.predict(X_test),y_test)
print(err1,err2,err3)
print(["SVR","Lasso","Linear regression"])
err1,err2,err3 = sklearn.metrics.r2_score(clf.predict(X_test),y_test),sklearn.metrics.r2_score(reg.predict(X_test),y_test),sklearn.metrics.r2_score(mod.predict(X_test),y_test)
print(err1,err2,err3)
Explanation: Summary
End of explanation
<END_TASK>