<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Diagnosing the data issues
Step1:  The data you'll be exloring
Step2:  Duplicated data
Step3:  So this is actually a mask. We can now ask for the data where the mask applies
Step4:  Missing data
Step5:  How much missing data do we have for heights?
Make a mask, with those who are missing, using isnull
Step6:  In python, False evaluates to 0, and True to 1. So we can count the number of missing by doing
Step7:  As before, we can use that mask on our original dataset, and see it
Step8:  How about age?
Step9:  And gender?
Here we're going to do something clever. We're going to get the value_counts, but we're going to change the parameter dropna (drop nulls) to false, so that we keep them. 
This would not be very useful with numerical data, but given that we know that age is categorical, we might as well
Step10:  But wait, we have another problem. We seem to have male and MALE
Step11:  Outliers
Step12:  This was useful, we can see that there are some really tall people, and some quite small ones. The distribution also looks close to normal
Who is outside of 2 standard deviations?
Let's make a quick function to deal with this...
Step13:  Looking at a few of these outliers
Step14:  Over
Step15:  Note
Step16:  How about the ages?
Step17:  Well, this is quite useless. The reason is that using standard deviations assumes that the distribution is normal, and we can clearly see it isn't. 
Let's try to solve this with other means...
What's the biggest outlier we have?
Step18:  What if we used percentiles?
Step19:  Note
Step20:  Now that looks a lot better. We can clearly identify that almost everyone is an adult, except the point on the extreme left.
Step21:  Starting to look better, what about the point on the far right?
<ASSISTANT_TASK:>
Python Code:
import pandas as pd 
import numpy as np 
% matplotlib inline
from matplotlib import pyplot as plt
Explanation: Diagnosing the data issues:
End of explanation
data = pd.read_csv('all_data.csv')
data.head(10)
Explanation: The data you'll be exloring:
End of explanation
duplicated_data = data.duplicated()
duplicated_data.head()
Explanation: Duplicated data:
We seem to have a problem with some duplicated data. We can find them using Pandas duplicated
End of explanation
data[duplicated_data]
Explanation: So this is actually a mask. We can now ask for the data where the mask applies:
End of explanation
heights = data['height']
ages = data['age']
gender = data['gender']
Explanation: Missing data:
End of explanation
missing_height = heights.isnull()
missing_height.head()
Explanation: How much missing data do we have for heights?
Make a mask, with those who are missing, using isnull
End of explanation
missing_height.sum()
Explanation: In python, False evaluates to 0, and True to 1. So we can count the number of missing by doing:
End of explanation
data[missing_height]
Explanation: As before, we can use that mask on our original dataset, and see it:
End of explanation
missing_ages = ages.isnull()
data[missing_ages]
Explanation: How about age?
End of explanation
gender.value_counts(dropna=False)
missing_gender = data['gender'].isnull()
data[missing_gender]
Explanation: And gender?
Here we're going to do something clever. We're going to get the value_counts, but we're going to change the parameter dropna (drop nulls) to false, so that we keep them. 
This would not be very useful with numerical data, but given that we know that age is categorical, we might as well:
End of explanation
gender.value_counts(dropna=False).plot(kind='bar', rot=0)
Explanation: But wait, we have another problem. We seem to have male and MALE:
End of explanation
heights.hist(bins=40, figsize=(16,4))
plt.xlabel('Height') 
plt.ylabel('Count')
Explanation: Outliers:
What is the distribution of the heights?
Note: pyplot is used here to make axis labels. Because otherwise, this happens
End of explanation
def print_analysis(series):
    for nr in range(1, 4):
        
        upper_limit = series.mean() + (nr * series.std())
        lower_limit = series.mean() - (nr * series.std())
                
        over_range = series > upper_limit
        percent_over_range = over_range.sum() / len(series) * 100
        
        under_range = series < lower_limit 
        percent_under_range = under_range.sum() / len(series) * 100
        
        in_range = (series < upper_limit) & (series > lower_limit)
        percent_in_range = in_range.sum() / len(series) * 100
        print('\nFor the range of %0.0f standard deviations:' % nr)
        print('  Lower limit:               %0.0f' % lower_limit)
        print('  Percent under range:       %0.1f%%' % percent_under_range)
        print('  Upper limit:               %0.0f' % upper_limit)
        print('  Percent over range:        %0.1f%%' % percent_over_range)
        print('  Percent within range:      %0.1f%%' % percent_in_range)
heights.hist(bins=20)
plt.xlabel('Height')
plt.ylabel('Count')
print_analysis(heights)
Explanation: This was useful, we can see that there are some really tall people, and some quite small ones. The distribution also looks close to normal
Who is outside of 2 standard deviations?
Let's make a quick function to deal with this...
End of explanation
heights[heights < heights.mean() - 2*heights.std()]
Explanation: Looking at a few of these outliers:
End of explanation
heights[heights > heights.mean() + 2*heights.std()]
Explanation: Over:
End of explanation
heights[heights < heights.mean() - 3*heights.std()]
heights[heights > heights.mean() + 3*heights.std()]
Explanation: Note: the 131cm and 208cm actually seem quite plausible. 
And outside 3 standard deviations?
Under:
End of explanation
ages.hist(bins=40, figsize=(16,4))
plt.xlabel('Age')
plt.ylabel('Count')
print_analysis(ages)
Explanation: How about the ages?
End of explanation
ages.max()
Explanation: Well, this is quite useless. The reason is that using standard deviations assumes that the distribution is normal, and we can clearly see it isn't. 
Let's try to solve this with other means...
What's the biggest outlier we have?
End of explanation
extreme_value = .99
ages.dropna().quantile(extreme_value)
Explanation: What if we used percentiles?
End of explanation
# under_extreme_value = ages, where ages is smaller than the extreme value: 
under_extreme_value = ages[ages < ages.dropna().quantile(extreme_value)]
under_extreme_value.hist(bins=40, figsize=(16,4))
plt.xlabel('Age')
plt.ylabel('Count')
Explanation: Note: we had to use .dropna() there, as otherwise pandas would raise a runtime error(try it!) 
Now, let's take a look at what our data is under this extreme value:
End of explanation
non_babies = under_extreme_value[under_extreme_value > 10]
non_babies.hist(bins=40, figsize=(16,4))
plt.xlabel('Age')
plt.ylabel('Count')
Explanation: Now that looks a lot better. We can clearly identify that almost everyone is an adult, except the point on the extreme left.
End of explanation
non_babies.max()
Explanation: Starting to look better, what about the point on the far right?
End of explanation
<END_TASK>