<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 California house price prediction
Load the data and explore it
Step1:  Create a test set
Step2:  Do stratified sampling
Assuming median_income is an important predictor, we need to categorize it. It is important to build categories such that there are a sufficient number of data points in each strata, else the stratum's importance is biased. To make sure, we need not too many strata (like it is now with median income) and strata are relatively wide.
Step3:  Now remove the income_cat field used for this sampling. We will learn on the median_income data instead
Step4:  Exploratory data analysis
Step5:  Do pairwise plot to understand how each feature is correlated to each other
Step6:  Focussing on relationship between income and house value
Step7:  Creating new features that are meaningful and also useful in prediction
Create the number of rooms per household, bedrooms per household, ratio of bedrooms to the rooms, number of people per household. We do this on the whole dataset, then collect the train and test datasets.
Step8:  Prepare data for ML
Step9:  Fill missing values using Imputer - using median values
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
housing = pd.read_csv(r"E:\GIS_Data\file_formats\CSV\housing.csv")
housing.head()
housing.info()
# find unique values in ocean proximity column
housing.ocean_proximity.value_counts()
#describe all numerical rows - basic stats
housing.describe()
%matplotlib inline
import matplotlib.pyplot as plt
housing.hist(bins=50, figsize=(20,15))
Explanation: California house price prediction
Load the data and explore it
End of explanation
from sklearn.model_selection import train_test_split
train_set, test_set = train_test_split(housing, test_size=0.2, random_state=42)
print(train_set.shape)
print(test_set.shape)
Explanation: Create a test set
End of explanation
# scale the median income down by dividing it by 1.5 and rounding up those which are greater than 5 to 5.0
import numpy as np
housing['income_cat'] = np.ceil(housing['median_income'] / 1.5) #up round to integers
#replace those with values > 5 with 5.0, values < 5 remain as is
housing['income_cat'].where(housing['income_cat'] < 5, 5.0, inplace=True)
housing['income_cat'].hist()
from sklearn.model_selection import StratifiedShuffleSplit
split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
for train_index, test_index in split.split(housing, housing['income_cat']):
    strat_train_set = housing.loc[train_index]
    strat_test_set = housing.loc[test_index]
Explanation: Do stratified sampling
Assuming median_income is an important predictor, we need to categorize it. It is important to build categories such that there are a sufficient number of data points in each strata, else the stratum's importance is biased. To make sure, we need not too many strata (like it is now with median income) and strata are relatively wide.
End of explanation
for _temp in (strat_test_set, strat_train_set):
    _temp.drop("income_cat", axis=1, inplace=True)
# Write the train and test data to disk
strat_test_set.to_csv('./housing_strat_test.csv')
strat_train_set.to_csv('./housing_strat_train.csv')
Explanation: Now remove the income_cat field used for this sampling. We will learn on the median_income data instead
End of explanation
strat_train_set.plot(kind='scatter', x='longitude', y='latitude', alpha=0.4, s=strat_train_set['population']/100,
                    label='population', figsize=(10,7), color=strat_train_set['median_house_value'], 
                     cmap=plt.get_cmap('jet'), colorbar=True)
plt.legend()
Explanation: Exploratory data analysis
End of explanation
import seaborn as sns
sns.pairplot(data=strat_train_set[['median_house_value','median_income','total_rooms','housing_median_age']])
Explanation: Do pairwise plot to understand how each feature is correlated to each other
End of explanation
strat_train_set.plot(kind='scatter', x='median_income', y='median_house_value', alpha=0.1)
Explanation: Focussing on relationship between income and house value
End of explanation
housing['rooms_per_household'] = housing['total_rooms'] / housing['households']
housing['bedrooms_per_household'] = housing['total_bedrooms'] / housing['households']
housing['bedrooms_per_rooms'] = housing['total_bedrooms'] / housing['total_rooms']
corr_matrix = housing.corr()
corr_matrix['median_house_value'].sort_values(ascending=False)
housing.plot(kind='scatter', x='bedrooms_per_household',y='median_house_value', alpha=0.5)
Explanation: Creating new features that are meaningful and also useful in prediction
Create the number of rooms per household, bedrooms per household, ratio of bedrooms to the rooms, number of people per household. We do this on the whole dataset, then collect the train and test datasets.
End of explanation
#create a copy without the house value column
housing = strat_train_set.drop('median_house_value', axis=1)
#create a copy of house value column into a new series, which will be the labeled data
housing_labels = strat_test_set['median_house_value'].copy()
Explanation: Prepare data for ML
End of explanation
from sklearn.preprocessing import Imputer
housing_imputer = Imputer(strategy='median')
#drop text columns let Imputer learn
housing_numeric = housing.drop('ocean_proximity', axis=1)
housing_imputer.fit(housing_numeric)
housing_imputer.statistics_
_x = housing_imputer.transform(housing_numeric)
housing_filled= pd.DataFrame(_x, columns=housing_numeric.columns)
housing_filled['ocean_proximity'] = housing['ocean_proximity']
housing_filled.head()
Explanation: Fill missing values using Imputer - using median values
End of explanation
<END_TASK>