<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <h1>Data setup</h1>
<h2>Use our function to read the data file</h2>
Step1:  <h1>Plotting data on google maps</h1>
<h2>gmplot library</h2>
https
Step2:  <h3>Our data dataframe contains latitudes and longitudes for each complaint.</h3>
<h3>We can draw a heatmap that will help us see the relative concentration of complaints using lats and lons</h3>
<h3>Set up the map</h3>
<h4>GoogleMapPlotter constructor</h4>
<ul>
<li>GoogleMapPlotter(center_lat, center_lng, zoom)
<li>from_geocode(location_string,zoom)
Step3:  <h3>Then generate the heatmap passing the two data series (latitude and longitude) to the function</h3>
Step4:  <h3>Save the heatmap to an html file</h3>
<h4>The html file can be viewed, printed, or included in another html page</h4>
Step5:  <h1>Let's do some grouping operations</h1>
<h2>Incidents by Borough</h2>
Step6:  <h2>Group data by borough and plot a bar chart of the incident count</h2>
Step7:  <h1>Incidents by Agency</h1>
Step8:  <h2>Let's combine the two in a single graph</h2>
Step9:  <h2>This is quite unreadable and pointless!</h2>
<h3>We can unstack the groups so that we get borough by agency</h3>
Step10:  <h3>Increase the size of the image and add a title</h3>
Step11:  <h1>Digression
Step12:  <h4>Group by country</h4>
Step13:  <h4>Group by multiple columns</h4>
Step14:  <h4>Group by age groups</h4>
Step15:  <h2>Grouping by the values in a column</h2>
<h3>For example, grouping the data by values in a column that are greater than or less than zero</h3>
Step16:  <h3>Write a function that takes three arguments - a dataframe, an index, and a column name and returns the grouping for that row</h3>
Step17:  <h2>Now we can compute stats on these groups</h2>
Step18:  <h1>Incidents by time</h1>
<p>We know the creation date of each incident so we can build a bar graph of number of incidents by month
<p>Not particularly useful with a few months data but if we had all data from 2010, we could use this sort of
analysis to eyeball trends and seasonality
<p>We're going to need to do some data manipulation for this
<h3>We'll start by creating a new date field yyyymm
Step19:  <h1>Examining agencies</h1>
<h2>We'll look at the frequency by agency and report the top 5 values</h2>
Step20:  <h3>We can drill down into complaints by Agency by borough</h3>
Step21:  <h3>We can create 'top 5 Agency' subplots subplots for each borough</h3>
Step22:  <h1>Processing time</h1>
<h2>We can compute simple statistics on processing time</h2>
Step23:  <h3>But it is easier to convert the timedelta processing_time into floats for calculation purposes</h3>
Step24:  <h2>Now we can compute stats easily</h2>
<ASSISTANT_TASK:>
Python Code:
def read_311_data(datafile):
    import pandas as pd
    import numpy as np
    
    #Add the fix_zip function
    def fix_zip(input_zip):
        try:
            input_zip = int(float(input_zip))
        except:
            try:
                input_zip = int(input_zip.split('-')[0])
            except:
                return np.NaN
        if input_zip < 10000 or input_zip > 19999:
            return np.NaN
        return str(input_zip)
    
    #Read the file
    df = pd.read_csv(datafile,index_col='Unique Key')
    
    #fix the zip
    df['Incident Zip'] = df['Incident Zip'].apply(fix_zip)
    
    #drop all rows that have any nans in them (note the easier syntax!)
    
    df = df.dropna(how='any')
    
    #get rid of unspecified boroughs
    df = df[df['Borough'] != 'Unspecified']
    
    #Convert times to datetime and create a processing time column
    
    import datetime
    df['Created Date'] = df['Created Date'].apply(lambda x:datetime.datetime.strptime(x,'%m/%d/%y %H:%M'))
    df['Closed Date'] = df['Closed Date'].apply(lambda x:datetime.datetime.strptime(x,'%m/%d/%y %H:%M'))
    df['processing_time'] =  df['Closed Date'] - df['Created Date']
    
    #Finally, get rid of negative processing times and return the final data frame
    
    df = df[df['processing_time']>=datetime.timedelta(0,0,0)]
    
    return df
    
datafile = "nyc_311_data_subset-2.csv"
data = read_311_data(datafile)
Explanation: <h1>Data setup</h1>
<h2>Use our function to read the data file</h2>
End of explanation
!pip install gmplot --upgrade
Explanation: <h1>Plotting data on google maps</h1>
<h2>gmplot library</h2>
https://github.com/vgm64/gmplot
End of explanation
import gmplot
#gmap = gmplot.GoogleMapPlotter(40.7128, -74.0059, 8)
gmap = gmplot.GoogleMapPlotter.from_geocode("New York",10)
Explanation: <h3>Our data dataframe contains latitudes and longitudes for each complaint.</h3>
<h3>We can draw a heatmap that will help us see the relative concentration of complaints using lats and lons</h3>
<h3>Set up the map</h3>
<h4>GoogleMapPlotter constructor</h4>
<ul>
<li>GoogleMapPlotter(center_lat, center_lng, zoom)
<li>from_geocode(location_string,zoom)
End of explanation
#Then generate a heatmap using the latitudes and longitudes
gmap.heatmap(data['Latitude'], data['Longitude'])
Explanation: <h3>Then generate the heatmap passing the two data series (latitude and longitude) to the function</h3>
End of explanation
gmap.draw('incidents3.html')
Explanation: <h3>Save the heatmap to an html file</h3>
<h4>The html file can be viewed, printed, or included in another html page</h4>
End of explanation
%matplotlib inline
Explanation: <h1>Let's do some grouping operations</h1>
<h2>Incidents by Borough</h2>
End of explanation
borough_group = data.groupby('Borough')
borough_group.size().plot(kind='bar')
#kind can be 'hist', 'scatter'
Explanation: <h2>Group data by borough and plot a bar chart of the incident count</h2>
End of explanation
agency_group = data.groupby('Agency')
agency_group.size().plot(kind='bar')
Explanation: <h1>Incidents by Agency</h1>
End of explanation
agency_borough = data.groupby(['Agency','Borough'])
agency_borough.size().plot(kind='bar')
Explanation: <h2>Let's combine the two in a single graph</h2>
End of explanation
agency_borough.size().unstack().plot(kind='bar')
Explanation: <h2>This is quite unreadable and pointless!</h2>
<h3>We can unstack the groups so that we get borough by agency</h3>
End of explanation
agency_borough = data.groupby(['Agency','Borough'])
agency_borough.size().unstack().plot(kind='bar',title="Incidents in each Agency by Borough",figsize=(15,15))
Explanation: <h3>Increase the size of the image and add a title</h3>
End of explanation
import pandas as pd
writers = pd.DataFrame({'Author':['George Orwell','John Steinbeck',
                                  'Pearl Buck','Agatha Christie'],
                        'Country':['UK','USA','USA','UK'],
                        'Gender':['M','M','F','F'],
                        'Age':[46,66,80,85]})
                        
writers
Explanation: <h1>Digression: The pandas groupby function</h1>
<h4>You can use functions to group data</h4>
End of explanation
grouped = writers.groupby('Country')
#grouped.first()
#grouped.last()
#grouped.sum()
#grouped.mean()
grouped.apply(sum)
grouped.groups
Explanation: <h4>Group by country</h4>
End of explanation
grouped = writers.groupby(['Country','Gender'])
grouped.groups
Explanation: <h4>Group by multiple columns</h4>
End of explanation
def age_groups(df,index,col):
    print(index,col)
    if df[col].iloc[index] < 30:
        return 'Young'
    if df[col].iloc[index] < 60:
        return 'Middle'
    else:
        return 'Old'
writers['Age'].iloc[0]
grouped = writers.groupby(lambda x: age_groups(writers,x,'Age'))
grouped.groups
Explanation: <h4>Group by age groups</h4>
End of explanation
import numpy as np
people = pd.DataFrame(np.random.randn(5, 5), columns=['a', 'b', 'c', 'd', 'e'], index=['Joe', 'Steve', 'Wes', 'Jim', 'Travis'])
people
Explanation: <h2>Grouping by the values in a column</h2>
<h3>For example, grouping the data by values in a column that are greater than or less than zero</h3>
End of explanation
def GroupColFunc(df, ind, col):
    if df[col].loc[ind] > 0:
        return 'Group1'
    else:
        return 'Group2'
people.groupby(lambda x: GroupColFunc(people, x, 'a')).groups
Explanation: <h3>Write a function that takes three arguments - a dataframe, an index, and a column name and returns the grouping for that row</h3>
End of explanation
print(people.groupby(lambda x: GroupColFunc(people, x, 'a')).mean())
print(people.groupby(lambda x: GroupColFunc(people, x, 'a')).std())
Explanation: <h2>Now we can compute stats on these groups</h2>
End of explanation
import datetime
data['yyyymm'] = data['Created Date'].apply(lambda x:datetime.datetime.strftime(x,'%Y%m'))
data['yyyymm']
date_agency = data.groupby(['yyyymm','Agency'])
date_agency.size().unstack().plot(kind='bar',figsize=(15,15))
Explanation: <h1>Incidents by time</h1>
<p>We know the creation date of each incident so we can build a bar graph of number of incidents by month
<p>Not particularly useful with a few months data but if we had all data from 2010, we could use this sort of
analysis to eyeball trends and seasonality
<p>We're going to need to do some data manipulation for this
<h3>We'll start by creating a new date field yyyymm
End of explanation
data.groupby('Agency').size().sort_values(ascending=False)
data.groupby('Agency').size().sort_values(ascending=False).plot(kind='bar', figsize=(20,4))
Explanation: <h1>Examining agencies</h1>
<h2>We'll look at the frequency by agency and report the top 5 values</h2>
End of explanation
agency_borough = data.groupby(['Agency', 'Borough']).size().unstack()
agency_borough
Explanation: <h3>We can drill down into complaints by Agency by borough</h3>
End of explanation
#We'll arrange the subplots in two rows and three columns. 
#Since we have only 5 boroughs, one plot will be blank
COL_NUM = 2
ROW_NUM = 3
import matplotlib.pyplot as plt
fig, axes = plt.subplots(ROW_NUM, COL_NUM, figsize=(12,12))
for i, (label, col) in enumerate(agency_borough.iteritems()): 
    ax = axes[int(i/COL_NUM), i%COL_NUM]
    col = col.sort_values(ascending=False)[:5] 
    col.plot(kind='barh', ax=ax)
    ax.set_title(label)
plt.tight_layout() 
for i, (label, col) in enumerate(agency_borough.iteritems()): 
    print(i,label,col)
Explanation: <h3>We can create 'top 5 Agency' subplots subplots for each borough</h3>
End of explanation
grouped = data[['processing_time','Borough']].groupby('Borough')
grouped.describe()
Explanation: <h1>Processing time</h1>
<h2>We can compute simple statistics on processing time</h2>
End of explanation
import numpy as np
#The time it takes to process. Cleaned up
data['float_time'] =data['processing_time'].apply(lambda x:x/np.timedelta64(1, 'D'))
data
Explanation: <h3>But it is easier to convert the timedelta processing_time into floats for calculation purposes</h3>
End of explanation
grouped = data[['float_time','Agency']].groupby('Agency')
grouped.mean().sort_values('float_time',ascending=False)
data['float_time'].hist(bins=50)
Explanation: <h2>Now we can compute stats easily</h2>
End of explanation
<END_TASK>