<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Intro analysis to the dataset <a class="tocSkip">
Providing a simple visualization and classic statistical data analysis.
Imports
Import dependencies
Step1:  Import data
Step2:  Cleaning
Data ranges
Step3:  Observations
Step4:  Prune outliers (similar approaches
Step5:  EDA
Distributions
Step6:  Conclusions on cleaned data without normalization <a class="tocSkip">
Most histograms appear to be unimodal, most of them symmetric with the last 2 being left-skewed.
Scatterplot clusters
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
import matplotlib.style as style
style.use('ggplot')
#print(style.available)
import matplotlib.pyplot as plt
%matplotlib inline
import csv
import datetime
from IPython.core.display import display, HTML
Explanation: Intro analysis to the dataset <a class="tocSkip">
Providing a simple visualization and classic statistical data analysis.
Imports
Import dependencies
End of explanation
data = pd.read_csv("train.csv", header = 0)
data.head()
Explanation: Import data
End of explanation
print("Dataset length: %i" % len(data))
# ---
print("vendor_id: [%s, %s]" % (min(data["vendor_id"]), max(data["vendor_id"])))
print(" ")
print("pickup_datetime: \t[%s, %s]" % (min(data["pickup_datetime"]), max(data["pickup_datetime"])))
print("dropoff_datetime: \t[%s, %s]" % (min(data["dropoff_datetime"]), max(data["dropoff_datetime"])))
print(" ")
print("pickup_longitude: \t[%s, %s]" % (min(data["pickup_longitude"]), max(data["pickup_longitude"])))
print("pickup_latitude: \t[%s, %s]" % (min(data["pickup_latitude"]), max(data["pickup_latitude"])))
print(" ")
print("dropoff_longitude: \t[%s, %s]" % (min(data["dropoff_longitude"]), max(data["dropoff_longitude"])))
print("dropoff_latitude: \t[%s, %s]" % (min(data["dropoff_latitude"]), max(data["dropoff_latitude"])))
print(" ")
print("store_and_fwd_flag: [%s, %s]" % (min(data["store_and_fwd_flag"]), max(data["store_and_fwd_flag"])))
print("passenger_count: [%s, %s]" % (min(data["passenger_count"]), max(data["passenger_count"])))
max_trip_d = max(data["trip_duration"])
m, s = divmod(max_trip_d, 60)
h, m = divmod(m, 60)
max_trip_d_f = "%d:%02d:%02d" % (h, m, s)
print("trip_duration: [%s, %s]" % (min(data["trip_duration"]), max_trip_d_f))
print("---max (seconds): ", max_trip_d)
Explanation: Cleaning
Data ranges
End of explanation
#print(data.columns.values)
print("vendor_id: %s" % data["vendor_id"].dtypes)
print("pickup_datetime:  %s" % data["pickup_datetime"].dtypes)
print("dropoff_datetime: %s" % data["dropoff_datetime"].dtypes)
print()
print("pickup_longitude:  %s" % data["pickup_longitude"].dtypes)
print("pickup_latitude:   %s" % data["pickup_latitude"].dtypes)
print("dropoff_longitude: %s" % data["dropoff_longitude"].dtypes)
print("dropoff_latitude:  %s" % data["dropoff_latitude"].dtypes)
print()
print("store_and_fwd_flag: %s" % data["store_and_fwd_flag"].dtypes)
print("passenger_count:    %s" % data["passenger_count"].dtypes)
print("trip_duration:      %s" % data["trip_duration"].dtypes)
# resolution
data_clean = data
data_clean["pickup_datetime"]  = data_clean["pickup_datetime"].astype("datetime64")
data_clean["dropoff_datetime"] = data_clean["dropoff_datetime"].astype("datetime64")
Explanation: Observations:
+ Last maximum seems to be quite a big outlier.
Check if data types are appropriate.
End of explanation
# chosen as ~91 min (by analyzing the distribution below beforehand)
data_clean = data_clean[data.trip_duration < 5500]
Explanation: Prune outliers (similar approaches: trimming or winsorizing data)
End of explanation
display(HTML('<center><h1>Analyzing distribution for the series</h1></center>'))
data["pickup_datetime"].groupby(data["pickup_datetime"].dt.month).count().plot(kind="bar", rot=0)
data["dropoff_datetime"].groupby(data["dropoff_datetime"].dt.month).count().plot(kind="bar")
# initial
# xlim = [-80, -70]
# ylim = [40, 42]
xlim = [-74.2, -73.7]
ylim = [40.55, 40.95]
data_normalized = data_clean
data_normalized = data_normalized[(data_normalized.dropoff_latitude < ylim[1]) & (data_normalized.pickup_latitude < ylim[1])]
data_normalized = data_normalized[(data_normalized.dropoff_latitude > ylim[0]) & (data_normalized.pickup_latitude > ylim[0])]
data_normalized = data_normalized[(data_normalized.dropoff_longitude < xlim[1]) & (data_normalized.pickup_longitude < xlim[1])]
data_normalized = data_normalized[(data_normalized.dropoff_longitude > xlim[0]) & (data_normalized.pickup_longitude > xlim[0])]
data_normalized.hist(
    column=["pickup_longitude", "pickup_latitude", "dropoff_longitude", "dropoff_latitude"],
    figsize=(10, 10),
    weights = np.ones_like(data_normalized.index) / len(data_normalized.index),
    bins = 30
    #,sharey=True, sharex=True
)
display(HTML('<center><h1>Analyzing distribution for location (normalized)</h1></center>'))
data_clean.hist(
    column=["passenger_count", "trip_duration"], #"store_and_fwd_flag", 
    figsize=(10, 3)
)
plt.twiny()
display(HTML('<center><h1>Analyzing distribution for the other parameters</h1></center>'))
Explanation: EDA
Distributions
End of explanation
# some cleaning needed
# some big outliers are clogging the view
## values determined empirically with 0.05 marker plot below
xlim = [-74.2, -73.7] # -74.2, -73.85
ylim = [40.55, 40.95]
data_viz = data_clean
data_viz = data_viz[(data_viz.pickup_longitude > xlim[0]) & (data_viz.pickup_longitude < xlim[1])]
data_viz = data_viz[(data_viz.dropoff_longitude> xlim[0]) & (data_viz.dropoff_longitude< xlim[1])]
data_viz = data_viz[(data_viz.pickup_latitude  > ylim[0]) & (data_viz.pickup_latitude  < ylim[1])]
data_viz = data_viz[(data_viz.dropoff_latitude > ylim[0]) & (data_viz.dropoff_latitude < ylim[1])]
longitude = list(data_viz.pickup_longitude) + list(data_viz.dropoff_longitude)
latitude  = list(data_viz.pickup_latitude)  + list(data_viz.dropoff_latitude)
print("longitude: \t[%s, %s]" % (min(longitude), max(longitude)))
print("latitude: \t[%s, %s]" % (min(latitude), max(latitude)))
display(HTML('<center><h1>Scatter plot for points (estimating map)</h1></center>'))
plt.figure(figsize = (10,10))
plt.plot(longitude,latitude,'.', alpha = 0.4, markersize = 0.05)
        # 0.05 (less time, best for visualization) #10 (more time, best for outliers)
plt.show()
plt.figure(figsize = (10,5))
plt.subplots_adjust(wspace = 0.6) # wspace=None
# wspace = 0.2   # the amount of width reserved for blank space between subplots
display(HTML('<center><h1>Examining clusterization density (hex binning)</h1></center>'))
plt.subplot(121),\
    plt.hexbin(longitude,latitude, gridsize=50, cmap='inferno'),\
    plt.title('no log')
plt.colorbar().set_label('counts')
plt.subplot(122),\
    plt.hexbin(longitude,latitude, gridsize=50, bins='log', cmap='inferno'),\
    plt.title('Log'),\
    #plt.plot(longitude,latitude,'.', alpha = 0.1, c='c', markersize = 0.05)
    
plt.colorbar().set_label('log10(N)')
plt.show()
Explanation: Conclusions on cleaned data without normalization <a class="tocSkip">
Most histograms appear to be unimodal, most of them symmetric with the last 2 being left-skewed.
Scatterplot clusters
End of explanation
<END_TASK>