<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Software Engineering for Data Scientists
Manipulating Data with Python
CSE 583
Today's Objectives
1. Opening & Navigating the Jupyter Notebook
2. Simple Math in the Jupyter Notebook
3. Loading data with pandas
4. Cleaning and Manipulating data with pandas
5. Visualizing data with pandas & matplotlib
1. Opening and Navigating the IPython Notebook
We will start today with the interactive environment that we will be using often through the course
Step1:  uncomment this to download the data
Step2:  Loading Data with Pandas
Step3:  Because we'll use it so much, we often import under a shortened name using the import ... as ... pattern
Step4:  Now we can use the read_csv command to read the comma-separated-value data
Step5:  Note
Step6:  The shape attribute shows us the number of elements
Step7:  The columns attribute gives us the column names
Step8:  The index attribute gives us the index names
Step9:  The dtypes attribute gives the data types of each column
Step10:  4. Manipulating data with pandas
Here we'll cover some key features of manipulating data with pandas
Access columns by name using square-bracket indexing
Step11:  Mathematical operations on columns happen element-wise
Step12:  Columns can be created (or overwritten) with the assignment operator.
Let's create a tripminutes column with the number of minutes for each trip
Step13:  Note that this manipulation only modifies the data frame in memory; if you want to save the modified dataframe to CSV, you can use the to_csv() method
Step14:  More complicated mathematical operations can be done with tools in the numpy package
Step15:  Working with Times
One trick to know when working with columns of times is that Pandas DateTimeIndex provides a nice interface for working with columns of times.
For a dataset of this size, using pd.to_datetime and specifying the date format can make things much faster (from the strftime reference, we see that the pronto data has format "%m/%d/%Y %I
Step16:  (Note
Step17:  Simple Grouping of Data
The real power of Pandas comes in its tools for grouping and aggregating data. Here we'll look at value counts and the basics of group-by operations.
Step18:  Value Counts
Pandas includes an array of useful functionality for manipulating and analyzing tabular data.
We'll take a look at two of these here.
The pandas.value_counts returns statistics on the unique values within each column.
We can use it, for example, to break down rides by gender
Step19:  Or to break down rides by age
Step20:  By default, the values rather than the index are sorted. Use sort=False to turn this behavior off
Step21:  We can explore other things as well
Step22:  Group-by Operation
One of the killer features of the Pandas dataframe is the ability to do group-by operations.
You can visualize the group-by like this (image borrowed from the Python Data Science Handbook)
Step23:  The simplest version of a groupby looks like this, and you can use almost any aggregation function you wish (mean, median, sum, minimum, maximum, standard deviation, count, etc.)
&lt;data object&gt;.groupby(&lt;grouping values&gt;).&lt;aggregate&gt;()
for example, we can group by gender and find the average of all numerical columns
Step24:  It's also possible to indes the grouped object like it is a dataframe
Step25:  You can even group by multiple values
Step26:  The unstack() operation can help make sense of this type of multiply-grouped data. What this technically does is split a multiple-valued index into an index plus columns
Step27:  5. Visualizing data with pandas
Of course, looking at tables of data is not very intuitive.
Fortunately Pandas has many useful plotting functions built-in, all of which make use of the matplotlib library to generate plots.
Whenever you do plotting in the IPython notebook, you will want to first run this magic command which configures the notebook to work well with plots
Step28:  Now we can simply call the plot() method of any series or dataframe to get a reasonable view of the data
Step29:  Adjusting the Plot Style
Matplotlib has a number of plot styles you can use. For example, if you like R you might use the 'ggplot' style.
I like the 'seaborn' style
Step30:  Other plot types
Pandas supports a range of other plotting types; you can find these by using the <TAB> autocomplete on the plot method
Step31:  If you'd like to adjust the x and y limits of the plot, you can use the set_xlim() and set_ylim() method of the resulting object
Step32:  Breakout
Step33:  Split this plot by gender. Do you see any seasonal ridership patterns by gender?
Step34:  Split this plot by user type. Do you see any seasonal ridership patterns by usertype?
Step35:  Repeat the above three steps, counting the number of rides by time of day rather thatn by month.
<ASSISTANT_TASK:>
Python Code:
!ls
Explanation: Software Engineering for Data Scientists
Manipulating Data with Python
CSE 583
Today's Objectives
1. Opening & Navigating the Jupyter Notebook
2. Simple Math in the Jupyter Notebook
3. Loading data with pandas
4. Cleaning and Manipulating data with pandas
5. Visualizing data with pandas & matplotlib
1. Opening and Navigating the IPython Notebook
We will start today with the interactive environment that we will be using often through the course: the Jupyter Notebook.
We will walk through the following steps together:
Download miniconda (be sure to get Version 3.6) and install it on your system (hopefully you have done this before coming to class)
   
Use the conda command-line tool to update your package listing and install the IPython notebook:
Update conda's listing of packages for your system:
   $ conda update conda
Install IPython notebook and all its requirements
   $ conda install jupyter notebook
Navigate to the directory containing the course material. For example:
$ cd ~/courses/CSE583/
You should see a number of files in the directory, including these:
$ ls
   ...
   Breakout-Simple-Math.ipynb
   Lecture-Python-And-Data.ipynb
   ...
Type jupyter notebook in the terminal to start the notebook
$ jupyter notebook
If everything has worked correctly, it should automatically launch your default browser
   
Click on Lecture-Python-And-Data.ipynb to open the notebook containing the content for this lecture.
With that, you're set up to use the Jupyter notebook!
2. Simple Math in the Jupyter Notebook
Now that we have the Jupyter notebook up and running, we're going to do a short breakout exploring some of the mathematical functionality that Python offers.
Please open Breakout-Simple-Math.ipynb, find a partner, and make your way through that notebook, typing and executing code along the way.
3. Loading data with pandas
With this simple Python computation experience under our belt, we can now move to doing some more interesting analysis.
Python's Data Science Ecosystem
In addition to Python's built-in modules like the math module we explored above, there are also many often-used third-party modules that are core tools for doing data science with Python.
Some of the most important ones are:
numpy: Numerical Python
Numpy is short for "Numerical Python", and contains tools for efficient manipulation of arrays of data.
If you have used other computational tools like IDL or MatLab, Numpy should feel very familiar.
scipy: Scientific Python
Scipy is short for "Scientific Python", and contains a wide range of functionality for accomplishing common scientific tasks, such as optimization/minimization, numerical integration, interpolation, and much more.
We will not look closely at Scipy today, but we will use its functionality later in the course.
pandas: Labeled Data Manipulation in Python
Pandas is short for "Panel Data", and contains tools for doing more advanced manipulation of labeled data in Python, in particular with a columnar data structure called a Data Frame.
If you've used the R statistical language (and in particular the so-called "Hadley Stack"), much of the functionality in Pandas should feel very familiar.
matplotlib: Visualization in Python
Matplotlib started out as a Matlab plotting clone in Python, and has grown from there in the 15 years since its creation. It is the most popular data visualization tool currently in the Python data world (though other recent packages are starting to encroach on its monopoly).
Installing Pandas & friends
Because the above packages are not included in Python itself, you need to install them separately. While it is possible to install these from source (compiling the C and/or Fortran code that does the heavy lifting under the hood) it is much easier to use a package manager like conda. All it takes is to run
$ conda install numpy scipy pandas matplotlib
and (so long as your conda setup is working) the packages will be downloaded and installed on your system.
Downloading the data
shell commands can be run from the notebook by preceding them with an exclamation point:
End of explanation
# !curl -o pronto.csv https://data.seattle.gov/api/views/tw7j-dfaw/rows.csv?accessType=DOWNLOAD
Explanation: uncomment this to download the data:
End of explanation
import pandas
Explanation: Loading Data with Pandas
End of explanation
import pandas as pd
Explanation: Because we'll use it so much, we often import under a shortened name using the import ... as ... pattern:
End of explanation
data = pd.read_csv('pronto.csv')
Explanation: Now we can use the read_csv command to read the comma-separated-value data:
End of explanation
data.head()
data.tail()
Explanation: Note: strings in Python can be defined either with double quotes or single quotes
Viewing Pandas Dataframes
The head() and tail() methods show us the first and last rows of the data
End of explanation
data.shape
Explanation: The shape attribute shows us the number of elements:
End of explanation
data.columns
Explanation: The columns attribute gives us the column names
End of explanation
data.index
Explanation: The index attribute gives us the index names
End of explanation
data.dtypes
Explanation: The dtypes attribute gives the data types of each column:
End of explanation
data.columns
data['tripduration']
Explanation: 4. Manipulating data with pandas
Here we'll cover some key features of manipulating data with pandas
Access columns by name using square-bracket indexing:
End of explanation
data['tripduration'] / 60
Explanation: Mathematical operations on columns happen element-wise:
End of explanation
data['tripminutes'] = data['tripduration'] / 60
data.head()
Explanation: Columns can be created (or overwritten) with the assignment operator.
Let's create a tripminutes column with the number of minutes for each trip
End of explanation
data.to_csv('pronto-new.csv')
!ls
Explanation: Note that this manipulation only modifies the data frame in memory; if you want to save the modified dataframe to CSV, you can use the to_csv() method:
End of explanation
import numpy as np
np.exp(data['tripminutes'])
Explanation: More complicated mathematical operations can be done with tools in the numpy package:
End of explanation
data['starttime'].head()
pd.to_datetime(data['starttime'].head())
times = pd.DatetimeIndex(data['starttime'].head())
times.dayofweek
data['starttime']
times = pd.DatetimeIndex(pd.to_datetime(data['starttime'], format="%m/%d/%Y %I:%M:%S %p"))
Explanation: Working with Times
One trick to know when working with columns of times is that Pandas DateTimeIndex provides a nice interface for working with columns of times.
For a dataset of this size, using pd.to_datetime and specifying the date format can make things much faster (from the strftime reference, we see that the pronto data has format "%m/%d/%Y %I:%M:%S %p"
End of explanation
times.dayofweek
times.month
times
Explanation: (Note: you can also use infer_datetime_format=True in most cases to automatically infer the correct format, though due to a bug it doesn't work when AM/PM are present)
With it, we can extract, the hour of the day, the day of the week, the month, and a wide range of other views of the time:
End of explanation
data.head()
Explanation: Simple Grouping of Data
The real power of Pandas comes in its tools for grouping and aggregating data. Here we'll look at value counts and the basics of group-by operations.
End of explanation
pd.value_counts(data['gender'])
Explanation: Value Counts
Pandas includes an array of useful functionality for manipulating and analyzing tabular data.
We'll take a look at two of these here.
The pandas.value_counts returns statistics on the unique values within each column.
We can use it, for example, to break down rides by gender:
End of explanation
pd.value_counts(data['birthyear'])
Explanation: Or to break down rides by age:
End of explanation
pd.value_counts(data['birthyear'], sort=False)
Explanation: By default, the values rather than the index are sorted. Use sort=False to turn this behavior off:
End of explanation
pd.value_counts(times.dayofweek, sort=False)
pd.value_counts(times.month, sort=False)
pd.value_counts(data['gender'], dropna=False)
Explanation: We can explore other things as well: day of week, hour of day, etc.
End of explanation
from IPython.display import Image
Image('split_apply_combine.png')
Explanation: Group-by Operation
One of the killer features of the Pandas dataframe is the ability to do group-by operations.
You can visualize the group-by like this (image borrowed from the Python Data Science Handbook)
End of explanation
data.groupby('gender').count()
data.groupby('gender').mean()
Explanation: The simplest version of a groupby looks like this, and you can use almost any aggregation function you wish (mean, median, sum, minimum, maximum, standard deviation, count, etc.)
&lt;data object&gt;.groupby(&lt;grouping values&gt;).&lt;aggregate&gt;()
for example, we can group by gender and find the average of all numerical columns:
End of explanation
data.groupby('gender')['tripminutes'].mean()
Explanation: It's also possible to indes the grouped object like it is a dataframe:
End of explanation
data.groupby([times.hour, 'gender'])['tripminutes'].mean()
Explanation: You can even group by multiple values: for example we can look at the trip duration by time of day and by gender:
End of explanation
grouped = data.groupby([times.hour, 'gender'])['tripminutes'].mean().unstack()
grouped
Explanation: The unstack() operation can help make sense of this type of multiply-grouped data. What this technically does is split a multiple-valued index into an index plus columns:
End of explanation
%matplotlib inline
Explanation: 5. Visualizing data with pandas
Of course, looking at tables of data is not very intuitive.
Fortunately Pandas has many useful plotting functions built-in, all of which make use of the matplotlib library to generate plots.
Whenever you do plotting in the IPython notebook, you will want to first run this magic command which configures the notebook to work well with plots:
End of explanation
grouped.plot()
Explanation: Now we can simply call the plot() method of any series or dataframe to get a reasonable view of the data:
End of explanation
import matplotlib.pyplot as plt
plt.style.use('seaborn')
grouped.plot()
Explanation: Adjusting the Plot Style
Matplotlib has a number of plot styles you can use. For example, if you like R you might use the 'ggplot' style.
I like the 'seaborn' style:
End of explanation
grouped.plot.bar()
Explanation: Other plot types
Pandas supports a range of other plotting types; you can find these by using the <TAB> autocomplete on the plot method:
For example, we can create a histogram of trip durations:
End of explanation
ax = grouped.plot.bar()
ax.set_xlim(-1, 10)
Explanation: If you'd like to adjust the x and y limits of the plot, you can use the set_xlim() and set_ylim() method of the resulting object:
End of explanation
data['month'] = times.month
ax = data.groupby('month')['trip_id'].count().plot.bar();
Explanation: Breakout: Exploring the Data
Make a plot of the total number of rides as a function of month of the year (You'll need to extract the month, use a groupby, and find the appropriate aggregation to count the number in each group).
End of explanation
data.groupby(['month','gender'])['trip_id'].count().unstack().plot();
Explanation: Split this plot by gender. Do you see any seasonal ridership patterns by gender?
End of explanation
data.groupby(['month','usertype'])['trip_id'].count().unstack().plot();
Explanation: Split this plot by user type. Do you see any seasonal ridership patterns by usertype?
End of explanation
data['hour'] = times.month
ax = data.groupby('hour')['trip_id'].count().plot.bar();
data.groupby(['hour','gender'])['trip_id'].count().unstack().plot();
data.groupby(['hour','usertype'])['trip_id'].count().unstack().plot();
Explanation: Repeat the above three steps, counting the number of rides by time of day rather thatn by month.
End of explanation
<END_TASK>