<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <table>
<tr>
<td width=15%><img src="./img/UGA.png"></img></td>
<td><center><h1>Introduction to Python for Data Sciences</h1></center></td>
<td width=15%><a href="http
Step1:  Linear regression will obviously be a bad fit.
Step2:  Let us transform it into a 3-degree polynomial fit and perform the same linear regression.
Step3:  Pipeline
This 2-step fitting (Polynomial transform + Linear regression) calls for a replicated dataset which can be costly. That is why Scikit Learn implement a pipeline method that allows to perform multiple fit/transform sequentially.
This pipeline can then be used as a model.
Step4:  Validation and Hyperparameters tuning
We saw above (see the lasso example of the regression part) some basic examples of how to
Step5:  Grid Search
Now that scoring and cross validation is done, we can focus on investigating the best parameters of our polynomial model
Step6:  This enables to see the parameters corresponding to the quantities to fit
Step7:  We can then get the best parameters and the corresponding model.
Step8:  We notice that the grid search based on cross validation helped discarded overfitted models (as they were bad on validation sets).
Text and Image Features
We already saw an example of feature extraction from categorical data. However, for some particular categorical data, dedicated tools exist. For instance, text and images.
Text feature extraction
In Learning applications, words are usually more important than letters, so a basic way to extract features is to construct one feature per present word and count the occurences of this word. This is known as word count. An approach to mitigate very present words (like "the" , "a" , etc) is term frequency inverse document frequency (tf-idf) which weights the occurence count by how often it appears.
Step9:  The vectorizer has registered the feature names and outed a sparse matrix that can be converted to a Dataframe.
Step10:  The tf-idf verctorizer works the same way.
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
#import seaborn as sns
#sns.set() 
N = 100  #points to generate
X = np.sort(10*np.random.rand(N, 1)**0.8 , axis=0)  #abscisses
y = 4 + 0.4*np.random.rand(N) - 1. / (X.ravel() + 0.5)**2  - 1. / (10.5 - X.ravel() ) # some complicated function
plt.scatter(X,y)
Explanation: <table>
<tr>
<td width=15%><img src="./img/UGA.png"></img></td>
<td><center><h1>Introduction to Python for Data Sciences</h1></center></td>
<td width=15%><a href="http://www.iutzeler.org" style="font-size: 16px; font-weight: bold">Franck Iutzeler</a> </td>
</tr>
</table>
<br/><br/>
<center><a style="font-size: 40pt; font-weight: bold">Chap. 4 - Scikit Learn </a></center> 
<br/><br/>
4. Going Further
Creating new features/models
We saw above how to transform categorical features. It is possible to modify them in a number of ways in order to create different model.
For instance, from 1D  point/value couples $(x,y)$, the linear regression fits a line. However, if we tranform $x$ into $(x^1,x^2,x^3)$, the same linear regression will fit a 3-degree polynomial.
End of explanation
from sklearn.linear_model import LinearRegression
model = LinearRegression().fit(X, y)
yfit = model.predict(X)
plt.scatter(X, y)
plt.plot(X, yfit,color='r',label="linear regression")
plt.legend()
Explanation: Linear regression will obviously be a bad fit.
End of explanation
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=3, include_bias=False) # 3 degree without degree 0 (no constant)
XPoly = poly.fit_transform(X)
print(XPoly[:5,])
modelPoly = LinearRegression().fit(XPoly, y)
yfitPoly = modelPoly.predict(XPoly)
plt.scatter(X, y)
plt.plot(X, yfit,color='r',label="linear regression")
plt.plot(X, yfitPoly,color='k',label="Polynomial regression (deg 3)")
plt.legend(loc = 'lower right')
Explanation: Let us transform it into a 3-degree polynomial fit and perform the same linear regression.
End of explanation
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
polyFeat = PolynomialFeatures(degree=3, include_bias=False)
linReg = LinearRegression()
polyReg = Pipeline([ ('polyFeat',polyFeat) , ('linReg',linReg) ])
polyReg.fit(X, y) # X original not XPoly
yfitPolyNew = polyReg.predict(X)
plt.scatter(X, y)
plt.plot(X, yfit,color='r',label="linear regression")
plt.plot(X, yfitPolyNew,color='k',label="Polynomial regression (deg 3)")
plt.legend(loc = 'lower right')
Explanation: Pipeline
This 2-step fitting (Polynomial transform + Linear regression) calls for a replicated dataset which can be costly. That is why Scikit Learn implement a pipeline method that allows to perform multiple fit/transform sequentially.
This pipeline can then be used as a model.
End of explanation
from sklearn.model_selection import cross_val_score
cv_score = cross_val_score(polyReg, X, y, cv=5, scoring="neg_mean_absolute_error") # 5 groups cross validation
print(cv_score)
print("Mean score:" , np.mean(cv_score))
Explanation: Validation and Hyperparameters tuning
We saw above (see the lasso example of the regression part) some basic examples of how to:
* validate our model by splitting the dataset into training and testing set (using train_test_split)
* tune hyperparameter by looking at the error for different values
Scikit Learn actually provides some methods for that as well.
Validation
Scikit Learn offer a cross validation method that 
* split the dataset in several groups
* for each of these group, fit the model on all but this group and computer the error on this one
This way all the data has gone thought the learning and validating sets hence the cross validation. This is illustrated by the following figure from the  Python Data Science Handbook by Jake VanderPlas.
The score computer is computed either as the standard score of the estimator or can be precised with the <tt>scoring</tt> option (see the available metrics ). 
Warning All scorer objects follow the convention that higher return values are better than lower return values.
Let us compute the cross validation for our polynomial fit problem.
End of explanation
polyReg.get_params()
Explanation: Grid Search
Now that scoring and cross validation is done, we can focus on investigating the best parameters of our polynomial model:
* degree
* presence or not of an intercept
Let us see which are the parameters of our model (as this is a pipeline, this might be interesting to use the <tt>get_params</tt> function).
End of explanation
param_grid = [
  {'polyFeat__degree':  np.arange(1,12),
   'linReg__fit_intercept': [True,False],
   'polyFeat__include_bias': [True,False]}]
from sklearn.model_selection import GridSearchCV
grid = GridSearchCV(polyReg, param_grid, cv=5) 
grid.fit(X, y)
Explanation: This enables to see the parameters corresponding to the quantities to fit:
* degree: polyFeat__degree
* presence or not of an intercept: linReg__fit_intercept  and polyFeat__include_bias
We can now construct a dictionary of values to test.
End of explanation
grid.best_params_
best_model = grid.best_estimator_.fit(X,y)
overfit = polyReg.set_params(polyFeat__degree=15).fit(X,y)
Xplot = np.linspace(-1,10.5,100).reshape(-1, 1)
yBest = best_model.predict(Xplot)
yOver = overfit.predict(Xplot)
plt.scatter(X, y)
plt.plot(Xplot, yBest , 'r' ,label="Best polynomial")
plt.plot(Xplot, yOver , 'k' , label="overfitted (deg 15)")
plt.legend(loc = 'lower right')
plt.ylim([0,5])
plt.title("Best and overfitted models")
Explanation: We can then get the best parameters and the corresponding model.
End of explanation
f = open('./data/poems/poe-raven.txt', 'r')
poe = f.read().replace('\n',' ').replace('.','').replace(',','').replace('-','')
poe
from sklearn.feature_extraction.text import CountVectorizer
vec = CountVectorizer()
X = vec.fit_transform([poe])
X
Explanation: We notice that the grid search based on cross validation helped discarded overfitted models (as they were bad on validation sets).
Text and Image Features
We already saw an example of feature extraction from categorical data. However, for some particular categorical data, dedicated tools exist. For instance, text and images.
Text feature extraction
In Learning applications, words are usually more important than letters, so a basic way to extract features is to construct one feature per present word and count the occurences of this word. This is known as word count. An approach to mitigate very present words (like "the" , "a" , etc) is term frequency inverse document frequency (tf-idf) which weights the occurence count by how often it appears.
End of explanation
import pandas as pd
pd.DataFrame(X.toarray(), columns=vec.get_feature_names())
Explanation: The vectorizer has registered the feature names and outed a sparse matrix that can be converted to a Dataframe.
End of explanation
from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer()
X = vec.fit_transform([poe])
pd.DataFrame(X.toarray(), columns=vec.get_feature_names())
Explanation: The tf-idf verctorizer works the same way.
End of explanation
<END_TASK>