<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <center><h2>Scale your pandas workflows by changing one line of code</h2>
Exercise 2
Step1:  Dataset
Step2:  pandas.read_csv
Step3:  Expect pandas to take >3 minutes on EC2, longer locally
This is a good time to chat with your neighbor
Dicussion topics
- Do you work with a large amount of data daily?
- How big is your data?
- What’s the common use case of your data?
- Do you use any big data analytics tools?
- Do you use any interactive analytics tool?
- What’s are some drawbacks of your current interative analytic tools today?
modin.pandas.read_csv
Step4:  Are they equals?
Step5:  Concept for exercise
Step6:  Are they equal?
<ASSISTANT_TASK:>
Python Code:
import modin.pandas as pd
import pandas
import time
import modin.config as cfg
cfg.StorageFormat.put("omnisci")
Explanation: <center><h2>Scale your pandas workflows by changing one line of code</h2>
Exercise 2: Speed improvements
GOAL: Learn about common functionality that Modin speeds up by using all of your machine's cores.
Concept for Exercise: read_csv speedups
The most commonly used data ingestion method used in pandas is CSV files (link to pandas survey). This concept is designed to give an idea of the kinds of speedups possible, even on a non-distributed filesystem. Modin also supports other file formats for parallel and distributed reads, which can be found in the documentation.
We will import both Modin and pandas so that the speedups are evident.
Note: Rerunning the read_csv cells many times may result in degraded performance, depending on the memory of the machine
End of explanation
# We download data locally because currently `OmnisciOnNative` doesn't support read files from s3 storage.
# Note that this may take a few minutes to download.
import urllib.request
url_path = "https://modin-test.s3.us-west-1.amazonaws.com/yellow_tripdata_2015-01.csv"
urllib.request.urlretrieve(url_path, "taxi.csv")
path = "taxi.csv"
Explanation: Dataset: 2015 NYC taxi trip data
We will be using a version of this data already in S3, originally posted in this blog post: https://matthewrocklin.com/blog/work/2017/01/12/dask-dataframes
Size: ~200MB
End of explanation
start = time.time()
pandas_df = pandas.read_csv(path, parse_dates=["tpep_pickup_datetime", "tpep_dropoff_datetime"])
end = time.time()
pandas_duration = end - start
print("Time to read with pandas: {} seconds".format(round(pandas_duration, 3)))
Explanation: pandas.read_csv
End of explanation
start = time.time()
modin_df = pd.read_csv(path, parse_dates=["tpep_pickup_datetime", "tpep_dropoff_datetime"])
end = time.time()
modin_duration = end - start
print("Time to read with Modin: {} seconds".format(round(modin_duration, 3)))
print("Modin is {}x faster than pandas at `read_csv`!".format(round(pandas_duration / modin_duration, 2)))
Explanation: Expect pandas to take >3 minutes on EC2, longer locally
This is a good time to chat with your neighbor
Dicussion topics
- Do you work with a large amount of data daily?
- How big is your data?
- What’s the common use case of your data?
- Do you use any big data analytics tools?
- Do you use any interactive analytics tool?
- What’s are some drawbacks of your current interative analytic tools today?
modin.pandas.read_csv
End of explanation
modin_df
pandas_df
Explanation: Are they equals?
End of explanation
start = time.time()
pandas_groupby = pandas_df.groupby(by="total_amount").count()
end = time.time()
pandas_duration = end - start
print("Time to groupby with pandas: {} seconds".format(round(pandas_duration, 3)))
start = time.time()
modin_groupby = modin_df.groupby(by="total_amount").count()
end = time.time()
modin_duration = end - start
print("Time to groupby with Modin: {} seconds".format(round(modin_duration, 3)))
print("Modin is {}x faster than pandas at `groupby`!".format(round(pandas_duration / modin_duration, 2)))
Explanation: Concept for exercise: Groupby and aggregate
In pandas, you can groupby and aggregate. We will groupby a column in the dataset and use count for our aggregate.
End of explanation
pandas_groupby
modin_groupby
Explanation: Are they equal?
End of explanation
<END_TASK>