<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 House Prices Estimator
Note
Step1:  Here we have to find the 'NaN' values and fill them with the mean. Probably it's not the best way to complete the info where we have empty values but at least we are keeping the same distribution. 
Average and standard deviation are not modified with this method.
Step2:  The 'SalePrice' has a skewed graph. We can stabilize it applying a logarithmic operation because we know that all the values are positive.
Step3:  There are a lot of features in this dataset so we are going to select only the most correlated features with the 'SalePrice'.
With the following query we can see that the ten first features have a good correlation. In case we want to change the number of correlated features to be retrieved, we'll create a function to customize that.
Step4:  KFold
As usual, it's going to use the 'KFold' to split the dataset in different buckets.
Step5:  We implement two methos to plot the PCA in case we need to visualize the information in a 2D graph. We'll need to reduce all the features to only one feature (component).
Step6:  Removing the 1% of the anomalies we can get an stable prediction but it's not sure. Probably it's going to help but this would be removed from the final calculation.
Step7:  Model
Two methods are created, one of them is for training and the other for showing the calculated metrics.
This is still under study so it can be modified in the future. Until now the best regressor has been the gradient boost.
Step8:  Running Models
Step9:  Adding Categorical
Step10:  Get Predictions
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
#load the files
train = pd.read_csv('input/train.csv')
test = pd.read_csv('input/test.csv')
data = pd.concat([train, test])
#size of training dataset
train_samples = train.shape[0]
test_samples = test.shape[0]
# remove the Id feature
data.drop(['Id'],1, inplace=True);
#data.describe()
Explanation: House Prices Estimator
Note: It's a competition from Kaggle.com and the input data was retrieved from there.
Data Analysis
End of explanation
datanum = data.select_dtypes([np.number])
datanum = datanum.fillna(datanum.dropna().mean())
Explanation: Here we have to find the 'NaN' values and fill them with the mean. Probably it's not the best way to complete the info where we have empty values but at least we are keeping the same distribution. 
Average and standard deviation are not modified with this method.
End of explanation
import matplotlib.pyplot as plt
%matplotlib inline
# Transforming to non-skewed SalePrice
data.SalePrice = data.SalePrice.apply(np.log)
data.SalePrice.hist(bins=50)
Explanation: The 'SalePrice' has a skewed graph. We can stabilize it applying a logarithmic operation because we know that all the values are positive.
End of explanation
# Correlation features
datanum.corr()['SalePrice'].drop('SalePrice').sort_values(ascending=False).head(10)
def getDataWithHighCorrFeatures(data, numberFeatures=10):
    high_corr_feat_names = data.corr()['SalePrice'].drop('SalePrice').sort_values(ascending=False).head(numberFeatures).axes[0].tolist()
    #high_corr_feat_names.remove('SalePrice')
    return data[high_corr_feat_names]
Explanation: There are a lot of features in this dataset so we are going to select only the most correlated features with the 'SalePrice'.
With the following query we can see that the ten first features have a good correlation. In case we want to change the number of correlated features to be retrieved, we'll create a function to customize that.
End of explanation
from sklearn.model_selection import KFold
kf = KFold(n_splits=10, random_state=13)#, shuffle=True)
print(kf)
Explanation: KFold
As usual, it's going to use the 'KFold' to split the dataset in different buckets.
End of explanation
#plotting PCA
from sklearn.decomposition import PCA
def getX_PCA(X):
    pca = PCA(n_components=1)
    return pca.fit(X).transform(X)
    
def plotPCA(X, y):
    pca = PCA(n_components=1)
    X_r = pca.fit(X).transform(X)
    plt.plot(X_r, y, 'x')
Explanation: We implement two methos to plot the PCA in case we need to visualize the information in a 2D graph. We'll need to reduce all the features to only one feature (component).
End of explanation
from sklearn.covariance import EllipticEnvelope
def removeAnomalies(X_train, y_train, verbose=False):
    # fit the model
    ee = EllipticEnvelope(contamination=0.01,
                          assume_centered=True,
                          random_state=13)
    ee.fit(X_train)
    pred = ee.predict(X_train)
    X_anom = X_train[pred != 1]
    y_anom = y_train[pred != 1]
    X_no_anom = X_train[pred == 1]
    y_no_anom = y_train[pred == 1]
    
    if (verbose):
        print("Number samples no anomalies: {}".format(X_no_anom.shape[0]))
    #after removing anomalies
    #plt.scatter(getX_PCA(X_no_anom), y_no_anom)
    #plt.scatter(getX_PCA(X_anom), y_anom)
    
    return X_no_anom, y_no_anom
def idxNotAnomalies(X):
    ee = EllipticEnvelope(contamination=0.01,
                          assume_centered=True,
                          random_state=13)
    ee.fit(X)
    pred = ee.predict(X)
    return [index[0] for index, x in np.ndenumerate(pred) if x == 1]
Explanation: Removing the 1% of the anomalies we can get an stable prediction but it's not sure. Probably it's going to help but this would be removed from the final calculation.
End of explanation
# Linear regression
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
def train(X_train, y_train, verbose=False):
    #lr = LinearRegression()
    import xgboost as xgb
    lr = xgb.XGBRegressor(max_depth=5, 
                          n_estimators=250,
                          min_child_weight=7,
                          n_jobs=4)
    #
    batch = 0
    for train_idx, val_idx in kf.split(X_train, y_train):
        X_t, X_v = X_train[train_idx], X_train[val_idx]
        y_t, y_v = y_train[train_idx], y_train[val_idx]
        #training
        lr.fit(X_t, y_t)
        #calculate costs
        t_error = mean_squared_error(y_t, lr.predict(X_t))**0.5
        v_error = mean_squared_error(y_v, lr.predict(X_v))**0.5
        if verbose:
            print("{}) Training error: {:.2f}  Validation error: {:.2f} Score: {:.2f}"
                  .format(batch, t_error, v_error, lr.score(X_v, y_v)))
        batch += 1
        
    return lr
def metrics(model, X, y, verbose=False):
    #Scores
    if verbose:
        print("Training score: {:.4f}".format(model.score(X, y)))
    #RMSLE
    #print(np.count_nonzero(~np.isfinite(model.predict(X))))
    rmsle = mean_squared_error(y, model.predict(X))**0.5
    if verbose:
        print("RMSLE: {:.4f}".format(rmsle))
        # Plotting the results
        plt.scatter(model.predict(X), y)
        
    return rmsle, model.score(X, y)
# Get polynomial features
from sklearn.preprocessing import PolynomialFeatures
def getPolynomial(X_train, X_no_anom, X_test):
    poly = PolynomialFeatures(degree=2)
    return poly.fit_transform(X_train), poly.fit_transform(X_no_anom), poly.fit_transform(X_test)
def getKeyWithMinError(X_train, X_no_amon, y_train, y_no_anom, verbose=False):
    rmsles = {}
    for f in range(1,X_train.shape[1]):
        model = train(X_no_anom[:,:f], y_no_anom, verbose=False)
        rmsles[f] = metrics(model, X_train[:,:f], y_train, verbose=False)
    min_error_key = min(rmsles, key=rmsles.get)
    
    if (verbose):
        print("Min error (k={}):{}".format(min_error_key, rmsles[min_error_key]))
    #model = train(X_train_pol[:,:min_error_key], y_train)
    #metrics(model, X_train_orig_pol[:,:min_error_key], y_train_orig)
    #pd.Series(rmsles).plot()
    return min_error_key
Explanation: Model
Two methods are created, one of them is for training and the other for showing the calculated metrics.
This is still under study so it can be modified in the future. Until now the best regressor has been the gradient boost.
End of explanation
import warnings
warnings.filterwarnings('ignore')
errors = []
for f in range(1,17):
    #print("====Corr feat: {}====".format(f))
    datanum_high_corr = getDataWithHighCorrFeatures(datanum, f)
    y = np.array(data['SalePrice'])
    X = np.array(datanum_high_corr)
    #split by idx
    idx = train_samples
    X_train, X_test = X[:idx], X[idx:]
    y_train, y_test = y[:idx], y[idx:]
    
    #print("Shape X train: {}".format(X_train.shape))
    X_no_anom, y_no_anom = removeAnomalies(X_train, y_train)
    #print("Shape X train (no anom): {}".format(X_no_anom.shape))
    X_train, X_no_anom, X_test = getPolynomial(X_train, X_no_anom, X_test)
    #print("Shape X train (poly): {}".format(X_no_anom.shape))
    key = 1000 #getKeyWithMinError(X_train, X_no_anom, y_train, y_no_anom)
    model = train(X_no_anom[:,:key], y_no_anom)
    error, score = metrics(model, X_train[:,:key], y_train)
    
    print("f:{} err:{:.3f} score:{:.3f}".format(f, error, score))
    errors.append(error)
# show graph
pd.Series(errors).plot()
Explanation: Running Models
End of explanation
features = data.select_dtypes([np.object]).axes[1].tolist()
features.append('SalePrice')
datacat = pd.get_dummies(data[features])
datacat = datacat.fillna(datacat.dropna().mean())
datacat.corr()['SalePrice'].drop('SalePrice').sort_values(ascending=False).head(10)
features_order_by_corr = datacat.corr()['SalePrice'].drop('SalePrice').sort_values(ascending=False).axes[0].tolist()
import warnings
warnings.filterwarnings('ignore')
datanum_high_corr = getDataWithHighCorrFeatures(datanum, 15)
Xn = np.array(datanum_high_corr)
#choosing the number of categorical
num_cat = 10
Xc = np.array(datacat[features_order_by_corr[:num_cat]])
y = np.array(data['SalePrice'])
poly = PolynomialFeatures(degree=2)
Xpn = poly.fit_transform(Xn)
X = np.concatenate([Xpn, Xc], axis=1)
no_anom_idx = idxNotAnomalies(Xn[:idx]) #only from numeric features
print(Xpn.shape)
print(Xc.shape)
print(X.shape)
#split by idx
idx = train_samples
X_train, X_test = X[:idx], X[idx:]
y_train, y_test = y[:idx], y[idx:]
print("Shape X train: {}".format(X_train.shape))
print("Shape X test: {}".format(X_test.shape))
print("Shape X train (no anom): {}".format(X_train[no_anom_idx].shape))
X_no_anom = X_train[no_anom_idx]
y_no_anom = y_train[no_anom_idx]
errors = {}
scores = {}
for f in range(15^2, X_train.shape[1]):
    modelt = train(X_no_anom[:,:f], y_no_anom, verbose=False)
    err, score = metrics(modelt, X_train[:,:f], y_train, verbose=False)
    if err > 1e-7:
        errors[f] = err
        scores[f] = score
    else:
        break
min_error_key = min(errors, key=errors.get)
max_score_key = max(scores, key=scores.get)
print("Min error: {:.3f}".format(errors[min_error_key]))
print("Max score: {:.3f}".format(scores[max_score_key]))
modelt = train(X_no_anom[:,:235], y_no_anom, verbose=False)
err, score = metrics(modelt, X_train[:,:235], y_train, verbose=False)
print(err)
print(score)
pd.Series(errors).plot()
pd.Series(scores).plot()
print(min_error_key)
Explanation: Adding Categorical
End of explanation
import os
from sklearn.model_selection import cross_val_score
from sklearn.metrics import make_scorer
min_error_key = 235
def RMSE(y_true,y_pred):
    rmse = mean_squared_error(y_true, y_pred)**0.5
    return rmse
modelt = train(X_no_anom[:,:min_error_key], y_no_anom, verbose=False)
err, score = metrics(modelt, X_train[:,:min_error_key], y_train, verbose=False)
print("Err: {:.3f} | R2: {:.3f}".format(err, score))
scores = cross_val_score(modelt, X_train[:,:min_error_key], y_train, 
                         scoring=make_scorer(RMSE, greater_is_better=True), cv=10)
print("Scores: {}".format(scores))
print("Score (mean): {:.3f}".format(scores.mean()))
predict = modelt.predict(X_test[:,:min_error_key])
#predictions are logs, return to the value
predict = np.exp(predict)
file = "Id,SalePrice" + os.linesep
startId = 1461
for i in range(len(X_test)):
    file += "{},{}".format(startId, (int)(predict[i])) + os.linesep
    startId += 1
# Save to file
with open('attempt.txt', 'w') as f:
    f.write(file)
# Using XGRegressor?
#lr = XGBRegressor(max_depth=5, n_estimators=250,min_child_weight=10)
Explanation: Get Predictions
End of explanation
<END_TASK>