<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Analysis of U.S. Incomes by Occupation and Gender
Notebook by Harish Kesava Rao
Use of this dataset should cite the Bureau of Labor Statistics as per their copyright information
Step1:  We assign more meaningful and self explanatory column headers. The names list can be assigned to the names parameter when reading in the csv file.
Step2:  To manipulate and perform operations on numerical columns, we will need to handle null/empty/non-numerical values in the numerical columns. In our dataset, this means everything except the Occupations column.
Step3:  The original header that was part of the csv can be dropped
Step4:  The following looks more readable
Step5:  Now, we will slice the dataset to understand more about the data
Step6:  The sorting did not happen by the All_workers column. Let's see why that is the case.
Step7:  The columns are objects. We will need to convert them to primitive types, such as integer, string etc.
Step8:  The conversion happened for All_workers, Male_Workers (in 1000s), Female_Workers (in 1000s), but not for the other columns. We will see why.
Step9:  The columns for which the type conversion did not happend contain 'Na' values in them. These look like NULL or NA values, but actually, they are strings with the value 'Na'. We can work around these values by replacing them with 0.
Step10:  This looks better, without the Na string values. Now, we will convert all the columns, except Occupations to int. 
Occupations is a string column and we can leave it as an object type.
Step11:  We will now slice the data and analyze the count of workers in each occupation.
Step12:  The first row shows the total number of workers. We will drop the row since it is not useful for our analysis and will skew the overall pattern.
Step13:  We have a plot showing top employee count for 5 occupations. We will append gender wise numbers on the same plot. For this purpose, we use pandas' melt function. https
Step14:  In the above plot, we see that there are some professions involving only male employees - such as pales, production, transportation, construction etc. Similarly, there are occuapations that involve only female employees - such as healthcare professionals, education etc. This is just a sample of 20 occupations. We can analyze more, as seen below.
Step15:  We can observe a similar trend in the above plot as well, where transportation, construction, production, maintenance and groundskeeping involve male employees and education and healthcare involves female employees. We can also see that business, office, sale etc. have both female and male employees. 
Now, let us sort the Occupations by weekly median income, irrespective of gender of employees
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
from pandas import DataFrame, Series
Explanation: Analysis of U.S. Incomes by Occupation and Gender
Notebook by Harish Kesava Rao
Use of this dataset should cite the Bureau of Labor Statistics as per their copyright information: The Bureau of Labor Statistics (BLS) is a Federal government agency and everything that we publish, both in hard copy and electronically, is in the public domain, except for previously copyrighted photographs and illustrations. You are free to use our public domain material without specific permission, although we do ask that you cite the Bureau of Labor Statistics as the source.
What kind of questions are we aiming to answer through the data analysis?
<ol>
<li>Is there an inequality of income in the labor force?<br></li>
<li>Is it more pronounced in certain sectors than others?<br></li>
<li>Can the dataset help us answer the questions?<br>
    a. Is the dataset in a useable format?<br>
    b. How much of pre-analysis preparation is required before we start analyzing the dataset?<br></li>
<li>What other inferences can we derive out of the dataset?</li>
</ol>
Analyzing the dataset
Median weekly earnings of full-time wage and salary workers by detailed occupation and sex.<br>
<b>Occupation:</b> Job title as given from BLS. Industry summaries are given in ALL CAPS.<br>
<b>All_workers:</b> Number of workers male and female, in thousands.<br>
<b>All_weekly:</b> Median weekly income including male and female workers, in USD.<br>
<b>M_workers:</b> Number of male workers, in thousands.<br>
<b>M_weekly:</b> Median weekly income for male workers, in USD.<br>
<b>F_workers:</b> Number of female workers, in thousands.<br>
<b>F_weekly:</b> Median weekly income for female workers, in USD.<br>
End of explanation
names = ['Occupations','All_workers', 'Weekly_Income_Overall', 
                             'Male_Workers (in 1000s)', 'Median_weekly_income (Male)',
                             'Female_Workers (in 1000s)', 'Median_weekly_income (Female)']
#mac
income = pd.read_csv('/Users/Harish/Documents/HK_Work/Python/Python-for-Data-Analysis/Kaggle-US-Incomes/inc_occ_gender.csv'
                   ,names=names, header=None)
#win
#income = pd.read_csv(r'C:\Users\\Documents\Personal\Python-for-Data-Analysis\Kaggle-US-Incomes\inc_occ_gender.csv'
                # ,names=names, header=None)
Explanation: We assign more meaningful and self explanatory column headers. The names list can be assigned to the names parameter when reading in the csv file.
End of explanation
income.head()
Explanation: To manipulate and perform operations on numerical columns, we will need to handle null/empty/non-numerical values in the numerical columns. In our dataset, this means everything except the Occupations column.
End of explanation
income.drop(0, inplace=True)
income['income_index'] = income.index
Explanation: The original header that was part of the csv can be dropped
End of explanation
income.head()
Explanation: The following looks more readable
End of explanation
#selecting just the occupations and worker count into a new dataframe
job_type_by_worker_count = income[['Occupations','All_workers']]
job_type_by_worker_count.sort_values(by='All_workers', ascending =False)
Explanation: Now, we will slice the dataset to understand more about the data
End of explanation
income.dtypes
Explanation: The sorting did not happen by the All_workers column. Let's see why that is the case.
End of explanation
job_type_by_worker_count2 = income.apply(pd.to_numeric, errors = 'ignore')
job_type_by_worker_count2.dtypes
Explanation: The columns are objects. We will need to convert them to primitive types, such as integer, string etc.
End of explanation
job_type_by_worker_count2.head(n=15)
Explanation: The conversion happened for All_workers, Male_Workers (in 1000s), Female_Workers (in 1000s), but not for the other columns. We will see why.
End of explanation
income.replace('Na',0,inplace=True)
income.head(n=15)
Explanation: The columns for which the type conversion did not happend contain 'Na' values in them. These look like NULL or NA values, but actually, they are strings with the value 'Na'. We can work around these values by replacing them with 0.
End of explanation
income_clean = income.apply(pd.to_numeric, errors = 'ignore')
income_clean.dtypes
Explanation: This looks better, without the Na string values. Now, we will convert all the columns, except Occupations to int. 
Occupations is a string column and we can leave it as an object type.
End of explanation
top10_worker_count = income_clean.sort_values(by='All_workers', axis=0, 
                                                           ascending=False)[['Occupations','All_workers',
                                                                                    'Male_Workers (in 1000s)',
                                                                                    'Female_Workers (in 1000s)']]
top10_worker_count.drop(1, axis=0, inplace=True)
Explanation: We will now slice the data and analyze the count of workers in each occupation.
End of explanation
import matplotlib
import matplotlib.pyplot as plt
import seaborn as sb
%matplotlib inline
top_worker_count_barplot = sb.factorplot(x='Occupations',
           y='All_workers', data=top10_worker_count.head(n=5), hue='All_workers', 
                                     size=7, aspect=2.3,kind='bar')
top_worker_count_barplot.set(xlabel = "Occupations", ylabel = "Workers - All, Male, Female", 
                             title = "Top worker counts and the occupations")
sb.set(font_scale=2.3)
Explanation: The first row shows the total number of workers. We will drop the row since it is not useful for our analysis and will skew the overall pattern.
End of explanation
top10_worker_count.head()
top10_worker_count = pd.melt(top10_worker_count, id_vars=["Occupations"], 
                 value_vars=['All_workers','Male_Workers (in 1000s)','Female_Workers (in 1000s)'])
top10_mf_count = income_clean.sort_values(by='All_workers', axis=0, 
                                                           ascending=False)[['Occupations',
                                                                                    'Male_Workers (in 1000s)',
                                                                                    'Female_Workers (in 1000s)']]
top10_mf_count.drop(1, axis=0, inplace=True)
top10_male_female = pd.melt(top10_mf_count, id_vars=["Occupations"], 
                 value_vars=['Male_Workers (in 1000s)','Female_Workers (in 1000s)'])
top10_worker_count.replace('NaN',0,inplace=True)
top10_male_female.replace('NaN',0,inplace=True)
sorted_worker_count_n = top10_worker_count.sort_values(by='value', axis=0, 
                                                           ascending=False)[['Occupations','variable','value']]
sorted_male_female_count_n = top10_male_female.sort_values(by='value', axis=0, 
                                                           ascending=False)[['Occupations','variable','value']]
sorted_worker_count_n.head(n=5)
sorted_male_female_count_n.head(n=5)
sorted_worker_count_n_barplot = sb.factorplot(x='Occupations',
           y='value', data=sorted_worker_count_n.head(n=10), hue='variable', 
                                         palette="bright",
                                     size=10, aspect=5,kind='bar')
sorted_worker_count_n_barplot.set(xlabel = "Occupations", ylabel = "Workers - All, Male, Female", 
                             title = "Top worker counts and the occupations")
sb.set(font_scale=4.0)
Explanation: We have a plot showing top employee count for 5 occupations. We will append gender wise numbers on the same plot. For this purpose, we use pandas' melt function. https://pandas.pydata.org/pandas-docs/stable/generated/pandas.melt.html
End of explanation
sorted_male_female_count_n_barplot = sb.factorplot(x='Occupations',
           y='value', data=sorted_male_female_count_n.head(n=10), hue='variable', 
                                         palette="bright",
                                     size=12, aspect=5,kind='bar')
sorted_male_female_count_n_barplot.set(xlabel = "Occupations", ylabel = "Workers - All, Male, Female", 
                             title = "Top male, female employee counts and the occupations")
sb.set(font_scale=4)
Explanation: In the above plot, we see that there are some professions involving only male employees - such as pales, production, transportation, construction etc. Similarly, there are occuapations that involve only female employees - such as healthcare professionals, education etc. This is just a sample of 20 occupations. We can analyze more, as seen below.
End of explanation
top10_income_count = income_clean.sort_values(by='Weekly_Income_Overall', axis=0, 
                                                           ascending=False)[['Occupations',
                                                                                    'Weekly_Income_Overall']]
low10_income_count = income_clean.sort_values(by='Weekly_Income_Overall', axis=0, 
                                                           ascending=True)[['Occupations',
                                                                                    'Weekly_Income_Overall']]
top10_income_count_barplot = sb.factorplot(x='Occupations',
           y='Weekly_Income_Overall', data=top10_income_count.head(n=5), hue='Occupations', 
                                         palette="bright",
                                     size=12, aspect=5, kind='bar')
top10_income_count_barplot.set(xlabel = "Occupations", ylabel = "Weekly Income", 
                             title = "Top Weekly Incomes and the respective occupations")
sb.set(font_scale=2)
#size=12, aspect=5,
low10_income_count = low10_income_count[(low10_income_count != 0).all(1)]
# lowest10_income_count_barplot = sb.factorplot(x='Weekly_Income_Overall',
#             y='Occupations', data=low10_income_count.head(n=10), hue='Occupations',palette="bright",
#                                              size=12, aspect=5, kind='bar')
# lowest10_income_count_barplot.set(xlabel='Occupations', ylabel = 'Weekly Income',
#                                  title = 'Lowest Weekly Incomes and the respective occupations')
Explanation: We can observe a similar trend in the above plot as well, where transportation, construction, production, maintenance and groundskeeping involve male employees and education and healthcare involves female employees. We can also see that business, office, sale etc. have both female and male employees. 
Now, let us sort the Occupations by weekly median income, irrespective of gender of employees
End of explanation
<END_TASK>