<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
Step1:  What does this data look like?
Step3:  What are the column names?
Step5:  What does a created and closed date look like?
Step7:  Okay, let's just save the created and closed dates.
Step8:  Exploring how to convert them into the length of time the complaints are open.
Step9:  Okay, let's make a list of how many minutes each one of these is open!
Step10:  201 or 1.7% of the items didn't have both a closed and created time
Step11:  And at least 1103 of the results are probably not accurate.
Step12:  Let's turn this into a Pandas DataFrame.
Step13:  Okay, let's make a histogram showing how many incidents happen.
Step14:  Looks like those 1103 0-minute cases are all outliers that are going to mess things up.
Measures of Center
Mean
Step15:  Median
The mean is a LOT higher than the median, which suggests the data is skewed.
Step16:  Mode
Step17:  Measures of Spread
Range
Step18:  Standard Deviation
Step19:  If you think outliers are numbers three standard deviations away from the mean, there are 68 outliers, but it doesn't catch all of the negative values.
Step20:  Interquartile Range
Step21:  If you think outliers are 1.5 times the interquartile range above the 3rd quartile or below the 1st quartile, there are 1067 outliers, more than with the standard deviation outliers, but still not getting rid of the negative values.
Step22:  Box and Whiskers
<ASSISTANT_TASK:>
Python Code:
cursor.execute(SELECT * 
    FROM dot_311 
    LIMIT 1)
cursor.fetchone()
Explanation: What does this data look like?
End of explanation
cursor.execute(SELECT column_name FROM information_schema.columns WHERE table_name='dot_311')
cursor.fetchall()
Explanation: What are the column names?
End of explanation
cursor.execute(SELECT created_date, closed_date 
    FROM dot_311 
    LIMIT 1)
cursor.fetchone()
Explanation: What does a created and closed date look like?
End of explanation
cursor.execute(SELECT created_date, closed_date 
    FROM dot_311)
results = cursor.fetchall()
Explanation: Okay, let's just save the created and closed dates.
End of explanation
results[0][0]
results[0][1]
results[0][1] - results[0][0]
dif = results[0][1] - results[0][0]
divmod(dif.days * 86400 + dif.seconds, 60)
# datetime.datetime.strftime(datetime.timedelta(0, 3780), "%H:%M")
dif = results[0][1] - results[0][0]
divmod(dif.seconds, 60)
dif = datetime.timedelta(2, 3780)
dif.days * 1440 + dif.seconds/60
results[:3]
timelen = []
for created, closed in results[:3]:
    print('created', created)
    print('closed', closed)
    print('')
Explanation: Exploring how to convert them into the length of time the complaints are open.
End of explanation
timelen = []
for created, closed in results:
    if created and closed:
        dif = closed - created
        timelen.append(dif.days * 1440 + dif.seconds/60)
Explanation: Okay, let's make a list of how many minutes each one of these is open!
End of explanation
len(results) - len(timelen)
201 / (len(results) + len(timelen))
Explanation: 201 or 1.7% of the items didn't have both a closed and created time
End of explanation
num = 0
for time in timelen:
    if time <= 0:
        num += 1
print(num)
sorted(timelen)[:20]
Explanation: And at least 1103 of the results are probably not accurate.
End of explanation
df = pd.DataFrame({'timelen': timelen})
Explanation: Let's turn this into a Pandas DataFrame.
End of explanation
df.hist(bins = 28, range = (-120, 1560))
Explanation: Okay, let's make a histogram showing how many incidents happen.
End of explanation
df.mean()
df[df['timelen'] >= 0].mean()
Explanation: Looks like those 1103 0-minute cases are all outliers that are going to mess things up.
Measures of Center
Mean
End of explanation
df.median()
df[df['timelen'] >= 0].median()
Explanation: Median
The mean is a LOT higher than the median, which suggests the data is skewed.
End of explanation
df.mode()
df[df['timelen'] >= 0].mode()
Explanation: Mode
End of explanation
max(timelen) - min(timelen)
df[df['timelen'] >= 0].max() - df[df['timelen'] >= 0].min()
Explanation: Measures of Spread
Range
End of explanation
df.std()
df[df['timelen'] >= 0].std()
Explanation: Standard Deviation
End of explanation
df.mean() + df.std() * 3
df.mean() - df.std() * 3
std_outliers = df[(df['timelen'] < -53111.819884) | (df['timelen'] > 72202.006197)]
len(std_outliers)
std_outliers.head()
df[df['timelen'] >= 0].mean() + df[df['timelen'] >= 0].std() * 3
df[df['timelen'] >= 0].mean() - df[df['timelen'] >= 0].std() * 3
std_outliers = df[df['timelen'] > 75447.537602]
len(std_outliers)
std_outliers.head()
Explanation: If you think outliers are numbers three standard deviations away from the mean, there are 68 outliers, but it doesn't catch all of the negative values.
End of explanation
IQR = df.quantile(q=0.75) - df.quantile(q=0.25)
IQR
IQR2 = df[df['timelen'] >= 0].quantile(q=0.75) - df[df['timelen'] >= 0].quantile(q=0.25)
IQR2
Explanation: Interquartile Range
End of explanation
df.quantile(q=0.25) - 1.5 * IQR
df.quantile(q=0.75) + 1.5 * IQR
IQR_outliers = df[(df['timelen'] < -8501.2125) | (df['timelen'] > 14366.020833)]
len(IQR_outliers)
IQR_outliers.head()
df[df['timelen'] >= 0].quantile(q=0.25) - 1.5 * IQR2
df[df['timelen'] >= 0].quantile(q=0.75) + 1.5 * IQR2
IQR_outliers2 = df[(df['timelen'] > 14366.020833)]
len(IQR_outliers2)
IQR_outliers2.head()
Explanation: If you think outliers are 1.5 times the interquartile range above the 3rd quartile or below the 1st quartile, there are 1067 outliers, more than with the standard deviation outliers, but still not getting rid of the negative values.
End of explanation
plt.boxplot(df['timelen'])
plt.boxplot(df[df['timelen'] >= 0]['timelen'])
Explanation: Box and Whiskers
End of explanation
<END_TASK>