<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Frame the problem
signal
Step1:  25%, 50%, 75% are all percentiles
Step2:  These are tail heavy
<ASSISTANT_TASK:>
Python Code:
import os
import pandas as pd
housing_file = os.path.expanduser('~/handson-ml/datasets/housing/housing.csv')
def load_housing_data(housing_path):
    return pd.read_csv(housing_path)
housing = load_housing_data(housing_file)
housing.head()
housing.info()
housing["ocean_proximity"].value_counts()
housing.describe()
Explanation: Frame the problem
signal: a piece of information fed into an ML system (see Shannon's information theory)
pipeline: a series of processes to process data
component: name of a processing unit in a pipeline
Select a performance measure
standard deviation (σ): square root of the variance, being the average of the squared deviation from the mean
root mean square error (RMSE): measures the standard deviation of errors made by a system's predictions
RMSE(X, h) = sqrt(1/m Σ(h(x) - y))
(See book for actual formula.)
m is the number of instances in dataset
x all features of the nth instance
y is the label for the nth instance
X is matrix of all feature values (except target)
h is the hypothosis: the system's prediction function
y^ is y-hat, the prediction value of h(x)
RMSE(X, h) is the cost function
generally preferred
MEA: Mean absolute error: also called (average absolute deviation) (see formula in book)
MAE(X, h) = 1/m Σ|h(x) - y|
RMSE corresponds to the Euclidean norm, or l2 norm, which measures distances with straight line, is usual geometry
MAE corresponds to Manhattan norm, or l1 norm, which measures distance only with orthogonal lines (as going through city blocks)
lk norm: see formula
l0 norm (of vector v): gives the cardinality of a vector (ie, number of elements)
l infinity: the maximum absolute value of a vector
The higher the norm index, the more it focuses on large values and not on smaller ones; RMSE is more sensitive to outliers than MAE. When outliers are exceptionally rare (as in a bell curve) the RMSE performs very well
Explore housing data
End of explanation
%matplotlib inline
import matplotlib.pylab as plt
housing.hist(bins=50, figsize=(20, 15))
plt.show()
Explanation: 25%, 50%, 75% are all percentiles: the percentile below which a values fall; also called quartiles (25% being first quartile)
End of explanation
# simple set aside 20%
import numpy as np
import numpy.random as rnd
def split_train_test_data(data, test_ratio):
    shuffled_indices = rand.permutation(len(data))  # also can give constant for repeated values
    set_test_size = int(len(data) * test_ratio)
    test_indices = shuffled_indices[:set_test_size]
    train_indices = shuffled_indices[set_test_size:]
    return data.iloc[train_indices], data.iloc[test_indices]
train_set, test_set = split_train_test_data(housing, 0.2)
print('len train set:', len(train_set))
print('len test set:', len(test_set))
# set aside 20 but ensure test always remains the same
import hashlib
def test_set_check(identifier, test_ratio, hash):
    return hash(identifier).digest()[-1] < 256 * test_ratio
def split_train_test_by_id(data, test_ratio, id_column, hash=hashlib.md5):
    ids = data[id_column]
    in_test_set = ids.apply(lambda id_: test_set_check(id_, test_ratio, hash))
    return data.loc[~in_test_set], data.loc[in_test_set]
# use row index as id
housing_with_id = housing.reset_index() # adds an 'index' column
train_set, test_set = split_train_test_by_id(housing_with_id, 0.2, "index")
# or just use sklearn cross validation
from sklearn.cross_validation import train_test_split
Explanation: These are tail heavy: extend farther to the right of the median than to the left
Create a test set
data snooping: looking at a test set
End of explanation
<END_TASK>