<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 01 - Example - Eliminating Outliers
This notebook presents how to eliminate the diagnosed outliers (in the previous Learning Unit).
By
Step1:  Load the dataset that will be used
Step2:  Let us drop the missing and duplicated values since they don't matter for now (already covered in other notebooks)
Step3:  Dealing with outliers
Time to deal with the issues previously found.
1) Delete observations - use feature bounds
The easiest way is to delete the observations (for when you know the bounds of your features). Let's use Age, since know the limits. Set the limits
Step4:  Create the mask
Step5:  Check if some outliers were caught
Step6:  Yes! Two were found! The mask_age variable contains the rows we want to keep, i.e., the rows that meet the bounds above. So, lets drop the above 2 rows
Step7:  2) Create classes/bins
Instead of having a range of values you can discretize in classes/bins. Make use of pandas' qcut
Step8:  Discretize!
Step9:  The limits of the defined classes/bins are
Step10:  We could replace the height values by the new five categories. Nevertheless, looks like a person with 252 cm is actually an outlier and the best would be to evaluate this value against two-standard deviations or percentile (e.g., 99%).  
Lets define the height bounds according to two-standard deviations from the mean.
3) Delete observations - use the standard deviation
Step11:  Which ones are out of the bounds?
Step12:  Let's delete these rows (mask_height contains the rows we want to keep)
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import numpy as np
% matplotlib inline
from matplotlib import pyplot as plt 
Explanation: 01 - Example - Eliminating Outliers
This notebook presents how to eliminate the diagnosed outliers (in the previous Learning Unit).
By: Hugo Lopes
Learning Unit 08
Some inital imports:
End of explanation
data = pd.read_csv('../data/data_with_problems.csv', index_col=0)
print('Our dataset has %d columns (features) and %d rows (people).' % (data.shape[1], data.shape[0]))
data.head(15)
Explanation: Load the dataset that will be used
End of explanation
data = data.drop_duplicates()
data = data.dropna()
print('Our dataset has %d columns (features) and %d rows (people).' % (data.shape[1], data.shape[0]))
Explanation: Let us drop the missing and duplicated values since they don't matter for now (already covered in other notebooks):
End of explanation
min_age = 0
max_age = 117 # oldest person currently alive
Explanation: Dealing with outliers
Time to deal with the issues previously found.
1) Delete observations - use feature bounds
The easiest way is to delete the observations (for when you know the bounds of your features). Let's use Age, since know the limits. Set the limits:
End of explanation
mask_age = (data['age'] >= min_age) & (data['age'] <= max_age)
mask_age.head(10)
Explanation: Create the mask:
End of explanation
data[~mask_age]
Explanation: Check if some outliers were caught:
End of explanation
data = data[mask_age]
print('Our dataset has %d columns (features) and %d rows (people).' % (data.shape[1], data.shape[0]))
Explanation: Yes! Two were found! The mask_age variable contains the rows we want to keep, i.e., the rows that meet the bounds above. So, lets drop the above 2 rows:
End of explanation
data['height'].hist(bins=100)
plt.title('Height population distribution')
plt.xlabel('cm')
plt.ylabel('freq')
plt.show()
Explanation: 2) Create classes/bins
Instead of having a range of values you can discretize in classes/bins. Make use of pandas' qcut: Discretize variable into equal-sized buckets.
End of explanation
height_bins = pd.qcut(data['height'], 
                      5, 
                      labels=['very short', 'short', 'average', 'tall', 'very tall'], 
                      retbins=True)
height_bins[0].head(10)
Explanation: Discretize!
End of explanation
height_bins[1]
Explanation: The limits of the defined classes/bins are:
End of explanation
# Calculate the mean and standard deviation
std_height = data['height'].std()
mean_height = data['height'].mean()
# The mask!
mask_height = (data['height'] > mean_height-2*std_height) & (data['height'] < mean_height+2*std_height)
print('Height bounds:')
print('> Minimum accepted height: %3.1f' % (mean_height-2*std_height))
print('> Maximum accepted height: %3.1f' % (mean_height+2*std_height))
Explanation: We could replace the height values by the new five categories. Nevertheless, looks like a person with 252 cm is actually an outlier and the best would be to evaluate this value against two-standard deviations or percentile (e.g., 99%).  
Lets define the height bounds according to two-standard deviations from the mean.
3) Delete observations - use the standard deviation
End of explanation
data.loc[~mask_height]
Explanation: Which ones are out of the bounds?
End of explanation
data = data[mask_height]
print('Our dataset has %d columns (features) and %d rows (people).' % (data.shape[1], data.shape[0]))
Explanation: Let's delete these rows (mask_height contains the rows we want to keep)
End of explanation
<END_TASK>