<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <p><font size="6"><b>Pandas
Step1:  Introduction
Step2:  Dates and times in pandas
The Timestamp object
Pandas has its own date and time objects, which are compatible with the standard datetime objects, but provide some more functionality to work with.  
The Timestamp object can also be constructed from a string
Step3:  Like with datetime.datetime objects, there are several useful attributes available on the Timestamp. For example, we can get the month
Step4:  Parsing datetime strings
Unfortunately, when working with real world data, you encounter many different datetime formats. Most of the time when you have to deal with them, they come in text format, e.g. from a CSV file. To work with those data in Pandas, we first have to parse the strings to actual Timestamp objects.
<div class="alert alert-info">
<b>REMEMBER</b>
Step5:  A detailed overview of how to specify the format string, see the table in the python documentation
Step6:  The to_datetime function can also be used to convert a full series of strings
Step7:  Notice the data type of this series
Step8:  To quickly construct some regular time series data, the pd.date_range function comes in handy
Step9:  Time series data
Step10:  We already know how to parse a date column with Pandas
Step11:  With set_index('datetime'), we set the column with datetime values as the index, which can be done by both Series and DataFrame.
Step12:  The steps above are provided as built-in functionality of read_csv
Step13:  <div class="alert alert-info">
<b>REMEMBER</b>
Step14:  Similar to a Series with datetime data, there are some attributes of the timestamp values available
Step15:  The plot method will also adapt it's labels (when you zoom in, you can see the different levels of detail of the datetime labels)
Step16:  We have to much data to sensibly plot on one figure. Let's see how we can easily select part of the data or aggregate the data to other time resolutions in the next sections.
Selecting data from a time series
We can use label based indexing on a timeseries as expected
Step17:  But, for convenience, indexing a time series also works with strings
Step18:  A nice feature is "partial string" indexing, where we can do implicit slicing by providing a partial datetime string.
E.g. all data of 2013
Step19:  Normally you would expect this to access a column named '2013', but as for a DatetimeIndex, pandas also tries to interprete it as a datetime slice.
Or all data of January up to March 2012
Step20:  <div class="alert alert-success">
<b>EXERCISE</b>
Step21:  <div class="alert alert-success">
<b>EXERCISE</b>
Step22:  <div class="alert alert-success">
<b>EXERCISE</b>
Step23:  <div class="alert alert-success">
<b>EXERCISE</b>
Step24:  The power of pandas
Step25:  <div class="alert alert-danger">
<b>NOTE</b>
Step26:  <div class="alert alert-info">
<b>REMEMBER</b>
Step27:  <div class="alert alert-success">
<b>EXERCISE</b>
Step28:  <div class="alert alert-success">
<b>EXERCISE</b>
Step29:  <div class="alert alert-success">
<b>EXERCISE</b>
Step30:  <div class="alert alert-success">
<b>EXERCISE</b>
Step31:  <div class="alert alert-success">
<b>EXERCISE</b>
<ASSISTANT_TASK:>
Python Code:
%matplotlib notebook
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
pd.options.display.max_rows = 8
Explanation: <p><font size="6"><b>Pandas: Working with time series data</b></font></p>
© 2016, Joris Van den Bossche and Stijn Van Hoey  (&#106;&#111;&#114;&#105;&#115;&#118;&#97;&#110;&#100;&#101;&#110;&#98;&#111;&#115;&#115;&#99;&#104;&#101;&#64;&#103;&#109;&#97;&#105;&#108;&#46;&#99;&#111;&#109;, &#115;&#116;&#105;&#106;&#110;&#118;&#97;&#110;&#104;&#111;&#101;&#121;&#64;&#103;&#109;&#97;&#105;&#108;&#46;&#99;&#111;&#109;). Licensed under CC BY 4.0 Creative Commons
End of explanation
import datetime
dt = datetime.datetime(year=2016, month=12, day=19, hour=13, minute=30)
dt
print(dt) # .day,...
print(dt.strftime("%d %B %Y"))
Explanation: Introduction: datetime module
Standard Python contains the datetime module to handle with date and time data:
End of explanation
ts = pd.Timestamp('2016-12-19')
ts
Explanation: Dates and times in pandas
The Timestamp object
Pandas has its own date and time objects, which are compatible with the standard datetime objects, but provide some more functionality to work with.  
The Timestamp object can also be constructed from a string:
End of explanation
ts.month
ts + pd.Timedelta('5 days')
Explanation: Like with datetime.datetime objects, there are several useful attributes available on the Timestamp. For example, we can get the month:
End of explanation
pd.to_datetime("2016-12-09")
pd.to_datetime("09/12/2016")
pd.to_datetime("09/12/2016", dayfirst=True)
pd.to_datetime("09/12/2016", format="%d/%m/%Y")
Explanation: Parsing datetime strings
Unfortunately, when working with real world data, you encounter many different datetime formats. Most of the time when you have to deal with them, they come in text format, e.g. from a CSV file. To work with those data in Pandas, we first have to parse the strings to actual Timestamp objects.
<div class="alert alert-info">
<b>REMEMBER</b>: <br><br>
from string formatted dates to Timestamp objects: `to_datetime` function
</div>
End of explanation
s = pd.Series(['2016-12-09 10:00:00', '2016-12-09, 11:00:00', '2016-12-09 12:00:00'])
Explanation: A detailed overview of how to specify the format string, see the table in the python documentation: https://docs.python.org/3.5/library/datetime.html#strftime-and-strptime-behavior
Timestamp data in a Series or DataFrame column
End of explanation
ts = pd.to_datetime(s)
ts
Explanation: The to_datetime function can also be used to convert a full series of strings:
End of explanation
ts.dt.hour
ts.dt.weekday
Explanation: Notice the data type of this series: the datetime64[ns] dtype. This indicates that we have a series of actual datetime values.
The same attributes as on single Timestamps are also available on a Series with datetime data, using the .dt accessor:
End of explanation
pd.Series(pd.date_range(start="2016-01-01", periods=10, freq='3H'))
Explanation: To quickly construct some regular time series data, the pd.date_range function comes in handy:
End of explanation
data = pd.read_csv("data/flowdata.csv")
data.head()
Explanation: Time series data: Timestamp in the index
River discharge example data
For the following demonstration of the time series functionality, we use a sample of discharge data of the Maarkebeek (Flanders) with 3 hour averaged values, derived from the Waterinfo website.
End of explanation
data['Time'] = pd.to_datetime(data['Time'])
Explanation: We already know how to parse a date column with Pandas:
End of explanation
data = data.set_index("Time")
data
Explanation: With set_index('datetime'), we set the column with datetime values as the index, which can be done by both Series and DataFrame.
End of explanation
data = pd.read_csv("data/flowdata.csv", index_col=0, parse_dates=True)
Explanation: The steps above are provided as built-in functionality of read_csv:
End of explanation
data.index
Explanation: <div class="alert alert-info">
<b>REMEMBER</b>: <br><br>
`pd.read_csv` provides a lot of built-in functionality to support this kind of transactions when reading in a file! Check the help of the read_csv function...
</div>
The DatetimeIndex
When we ensure the DataFrame has a DatetimeIndex, time-series related functionality becomes available:
End of explanation
data.index.day
data.index.dayofyear
data.index.year
Explanation: Similar to a Series with datetime data, there are some attributes of the timestamp values available:
End of explanation
data.plot()
Explanation: The plot method will also adapt it's labels (when you zoom in, you can see the different levels of detail of the datetime labels):
End of explanation
data[pd.Timestamp("2012-01-01 09:00"):pd.Timestamp("2012-01-01 19:00")]
Explanation: We have to much data to sensibly plot on one figure. Let's see how we can easily select part of the data or aggregate the data to other time resolutions in the next sections.
Selecting data from a time series
We can use label based indexing on a timeseries as expected:
End of explanation
data["2012-01-01 09:00":"2012-01-01 19:00"]
Explanation: But, for convenience, indexing a time series also works with strings:
End of explanation
data['2013']
Explanation: A nice feature is "partial string" indexing, where we can do implicit slicing by providing a partial datetime string.
E.g. all data of 2013:
End of explanation
data['2012-01':'2012-03']
Explanation: Normally you would expect this to access a column named '2013', but as for a DatetimeIndex, pandas also tries to interprete it as a datetime slice.
Or all data of January up to March 2012:
End of explanation
data['2012':]
Explanation: <div class="alert alert-success">
<b>EXERCISE</b>:
 <ul>
  <li>select all data starting from 2012</li>
</ul>
</div>
End of explanation
data[data.index.month == 1]
Explanation: <div class="alert alert-success">
<b>EXERCISE</b>:
 <ul>
  <li>select all data in January for all different years</li>
</ul>
</div>
End of explanation
data['months'] = data.index.month
data[data['months'].isin([1, 2, 3])]
data = data.drop("months", axis=1)
Explanation: <div class="alert alert-success">
<b>EXERCISE</b>:
 <ul>
  <li>select all data in January, February and March for all different years</li>
</ul>
</div>
End of explanation
data[(data.index.hour > 8) & (data.index.hour < 20)]
data.between_time('08:00', '20:00')
Explanation: <div class="alert alert-success">
<b>EXERCISE</b>:
 <ul>
  <li>select all 'daytime' data (between 8h and 20h) for all days</li>
</ul>
</div>
End of explanation
data.resample('D').mean().head()
Explanation: The power of pandas: resample
A very powerfull method is resample: converting the frequency of the time series (e.g. from hourly to daily data).
The time series has a frequency of 1 hour. I want to change this to daily:
End of explanation
data.resample('D').max().head()
Explanation: <div class="alert alert-danger">
<b>NOTE</b>:
 <ul>
  <li>with older versions of pandas, <code>data.resample('D').mean()</code> was expressed as <code>data.resample('D', how='mean')</code>.</li>
</ul>
</div>
Other mathematical methods can also be specified:
End of explanation
data.resample('A').mean().plot() # 10D
Explanation: <div class="alert alert-info">
<b>REMEMBER</b>: <br><br>
    The string to specify the new time frequency: http://pandas.pydata.org/pandas-docs/stable/timeseries.html#offset-aliases <br><br>
    These strings can also be combined with numbers, eg `'10D'`...
</div>
End of explanation
data.resample('M').std().plot() # 'A'
Explanation: <div class="alert alert-success">
<b>EXERCISE</b>:
 <ul>
  <li>plot the monthly standard deviation of the columns</li>
</ul>
</div>
End of explanation
fig, ax = plt.subplots()
data.loc['2011':'2012', 'L06_347'].resample('M').mean().plot(ax=ax)
data.loc['2011':'2012', 'L06_347'].resample('M').median().plot(ax=ax)
ax.legend(["mean", "median"])
data.loc['2011':'2012', 'L06_347'].resample('M').agg(['mean', 'median']).plot()
Explanation: <div class="alert alert-success">
<b>EXERCISE</b>:
 <ul>
  <li>plot the monthly mean and median values for the years 2011-2012 for 'L06_347'<br><br></li>
</ul>
    **Note** <br>You can create a new figure with `fig, ax = plt.subplots()` and add each of the plots to the created `ax` object (see documentation of pandas plot function)
</div>
End of explanation
daily = data['LS06_348'].resample('D').mean() # daily avergaes calculated
daily.resample('M').agg(['min', 'max']).plot() # monthly minimum and maximum values of these daily averages
Explanation: <div class="alert alert-success">
<b>EXERCISE</b>:
 <ul>
  <li>plot the monthly mininum and maximum daily average value of the 'LS06_348' column</li>
</ul>
</div>
End of explanation
fig, ax = plt.subplots()
data['2013'].mean().plot(kind='barh', ax=ax)
Explanation: <div class="alert alert-success">
<b>EXERCISE</b>:
 <ul>
  <li>make a bar plot of the mean of the stations in year of 2013 (Remark: create a `fig, ax = plt.subplots()` object and add the plot to the created ax</li>
</ul>
</div>
End of explanation
data.groupby(data.index.month).mean().plot()
Explanation: <div class="alert alert-success">
<b>EXERCISE</b>:
 <ul>
  <li>Calculate the typical yearly pattern with monthly resolution (plot of the typical monthly average over the years)</li>
</ul>
</div>
End of explanation
<END_TASK>