<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Performance
Texas state salaries
About 700,000 records. We'll use time to measure execution time. In the pydata video we used %timeit which takes the average of several runs. For this version to be displayed properly in github, I used %time.
Step1:  Pandas histogram
The built in histogram doesn't give us as much detail as we would like to...
Step2:  Let's try with seaborn's distplot.
Step3:  Ah yes. We have to do some data munging before we can use it, removing the NaN (not a number or null) values. That should also be taken into consideration when talking about performance, if you have to do extra steps.
Step4:  A little slower, nicer looking, but, again, not as informative as we'd like. Let's see how stem_graphic does.
Step5:  We can see a lot of detail. The extremes ARE extreme ($5.2M). I can also see the trend for non managerial, managerial, upper management and beyond (350K). And I can look at this for much large sets of data.
Make excel cry
12.5M rows, NYC yellow taxi trips data on a laptop (ran also on a full year and multi years using a dask cluster with hdfs storage - given enough nodes, you can have sub second response time).
Step6:  From the above results, we  see that the C written optimized word count (wc) application took close to 17 seconds to count words in the file. wc is used as a reference low watermark. It is expected to be faster than anything in Python loading that same document.
And sure enough, pandas took longer at around 29 seconds to load.
Step7:  loading the data (29s) and displaying it (less than 1.5s) took a little over 30 seconds. Less than 31s at any rate is quite acceptable for 12 million rows.
But we can do slightly better with dask (although, this scenario with a single local file and a laptop with only 4 core and limited memory and IO bandwidth doesn't fully demonstrate the real power of dask compared to a server or cluster and/or multiple files)
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import pandas as pd
import seaborn as sns
from stemgraphic import stem_graphic
texas = pd.read_csv('salaries.csv')
texas.describe(include='all')
Explanation: Performance
Texas state salaries
About 700,000 records. We'll use time to measure execution time. In the pydata video we used %timeit which takes the average of several runs. For this version to be displayed properly in github, I used %time.
End of explanation
%time ax = texas.Annual_salary.hist()
Explanation: Pandas histogram
The built in histogram doesn't give us as much detail as we would like to...
End of explanation
%time g = sns.distplot(texas.Annual_salary)
Explanation: Let's try with seaborn's distplot.
End of explanation
%time g = sns.distplot(texas.Annual_salary.dropna())
Explanation: Ah yes. We have to do some data munging before we can use it, removing the NaN (not a number or null) values. That should also be taken into consideration when talking about performance, if you have to do extra steps.
End of explanation
%time fig, ax = stem_graphic(texas.Annual_salary, display=500, random_state=1235)
Explanation: A little slower, nicer looking, but, again, not as informative as we'd like. Let's see how stem_graphic does.
End of explanation
!head yellow_tripdata_2015-01.csv
%time !wc yellow_tripdata_2015-01.csv
%time df1 = pd.read_csv('yellow_tripdata_2015-01.csv')
Explanation: We can see a lot of detail. The extremes ARE extreme ($5.2M). I can also see the trend for non managerial, managerial, upper management and beyond (350K). And I can look at this for much large sets of data.
Make excel cry
12.5M rows, NYC yellow taxi trips data on a laptop (ran also on a full year and multi years using a dask cluster with hdfs storage - given enough nodes, you can have sub second response time).
End of explanation
df1.head()
%%time
fig, ax = stem_graphic(df1.total_amount, display=500);
Explanation: From the above results, we  see that the C written optimized word count (wc) application took close to 17 seconds to count words in the file. wc is used as a reference low watermark. It is expected to be faster than anything in Python loading that same document.
And sure enough, pandas took longer at around 29 seconds to load.
End of explanation
# you need to have the dask module installed to run this part
import dask.multiprocessing
dask.set_options(get=dask.multiprocessing.get)
import dask.dataframe as dd
%%time
df = dd.read_csv('yellow_tripdata_2015-01.csv')
stem_graphic(df.total_amount, display=500)
Explanation: loading the data (29s) and displaying it (less than 1.5s) took a little over 30 seconds. Less than 31s at any rate is quite acceptable for 12 million rows.
But we can do slightly better with dask (although, this scenario with a single local file and a laptop with only 4 core and limited memory and IO bandwidth doesn't fully demonstrate the real power of dask compared to a server or cluster and/or multiple files)
End of explanation
<END_TASK>