<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Pythonic pandas
Using a tutorial about fast flexible pandas.
The pandas Python package is an effective way to examine and manipulate data.
The source code is available on Github, and be sure to check out pandas' library of extension modules.
Be careful when writing code for pandas, because Pythonic code may not necessarily be a good idea.
Like NumPy, pandas is designed for vectorized operations that replace explicit loops with array expressions.
This tutorial will attempt to demonstrate Pythonic pandas that will make the best use of the language and the library.
Our Task
The goal of this example will be to apply time-of-use energy tariffs to find the total cost of energy consumption for one year.
Make sure that you are up to speed with basic data selection and indexing.
Our problem is that at different hours of the day, the price for electricity varies, so the task is to multiply the electricity consumed for each hour by the correct price for the hour in which it was consumed.
Let’s read our data from a CSV file that has two columns
Step1:  The rows contains the electricity used in each hour for a one year period.
Each row indicates the usage for the hour starting at the specified time, so 1/1/13 0
Step2:  Both pandas and Numpy use the concept of dtypes as data types, and if no arguments are specified, date_time will take on an object dtype.
Step3:  This will be an issue with any column that can't neatly fit into a single data type.
Working with dates as strings is also an inefficient use of memory and programmer time (not to mention patience).
This exercise will work with time series data, and the date_time column will be formatted as an array of datetime objects called a pandas.Timestamp.
Step4:  If you're curious about alternatives to the code above, check out pandas.PeriodIndex, which can store ordinal values indicating regular time periods.
We now have a pandas.DataFrame called nrg that contains the data from our .csv file.
Notice how the time is displayed differently in the date_time column.
Step5:  Time for a timing decorator
The code above is pretty straightforward, but how fast does it run?
Let's find out by using a timing decorator called @timeit (an homage to Python's timeit).
This decorator behaves like timeit.repeat(), but it also allows you to return the result of the function itself as well as get the average runtime from multiple trials.
When you create a function and place the @timeit decorator above it, the function will be timed every time it is called.
Keep in mind that the decorator runs an outer and an inner loop.
Step6:  One easily overlooked detail is that the datetimes in the energy_consumption.csv file are not in ISO 8601 format.
You need YYYY-MM-DD HH
Step8:  However, our hourly costs depend on the time of day.
If you use a loop to do the conditional calculation, you are not using pandas the way it was intended.
For the rest of this tutorial, you'll start with a sub-optimal solution and work your way up to a Pythonic approach that leverages the full power of pandas.
Take a look at a loop approach and see how it performs using our timing decorator.
Step10:  Now for a computationally expensive and non-Pythonic loop
Step11:  You can consider the above to be an “antipattern” in pandas for several reasons.
First, initialize a list in which the outputs will be recorded.
Second, use the opaque object range(0, len(df)) to loop through nrg, then apply apply_rate(), and append the result to a list used to make the new DataFrame column.
Third, chained indexing with df.iloc[i]['date_time'] may lead to unintended results.
Each of these increase the time cost of the calculations.
On my machine, this loop took about 3 seconds for 8760 rows of data.
Next, you’ll look at some improved solutions for iteration over Pandas structures.
Looping with .itertuples() and .iterrows()
Instead of looping through a range of objects, you can use generator methods that yield one row at a time.
.itertuples() yields a namedtuple() for each row, with the row’s index value as the first element of the tuple.
A namedtuple() is a data structure from Python’s collections module that behaves like a Python tuple but has fields accessible by attribute lookup.
.iterrows() yields pairs (tuples) of (index, Series) for each row in the DataFrame.
While .itertuples() tends to be a bit faster, let’s focus on pandas and use .iterrows() in this example.
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
pd.__version__
nrg = pd.read_csv('energy_consumption.csv'); nrg.describe(include='all')
Explanation: Pythonic pandas
Using a tutorial about fast flexible pandas.
The pandas Python package is an effective way to examine and manipulate data.
The source code is available on Github, and be sure to check out pandas' library of extension modules.
Be careful when writing code for pandas, because Pythonic code may not necessarily be a good idea.
Like NumPy, pandas is designed for vectorized operations that replace explicit loops with array expressions.
This tutorial will attempt to demonstrate Pythonic pandas that will make the best use of the language and the library.
Our Task
The goal of this example will be to apply time-of-use energy tariffs to find the total cost of energy consumption for one year.
Make sure that you are up to speed with basic data selection and indexing.
Our problem is that at different hours of the day, the price for electricity varies, so the task is to multiply the electricity consumed for each hour by the correct price for the hour in which it was consumed.
Let’s read our data from a CSV file that has two columns: one for date plus time and one for electrical energy consumed in kilowatt hours (kWh):
End of explanation
nrg.head()
Explanation: The rows contains the electricity used in each hour for a one year period.
Each row indicates the usage for the hour starting at the specified time, so 1/1/13 0:00 indicates the usage for the first hour of January 1st.
Working with datetime data
Let's take a closer look at our data:
End of explanation
nrg.dtypes
# https://docs.python.org/3/library/functions.html#type
# https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.iat.html
type(nrg.iat[0,0])
Explanation: Both pandas and Numpy use the concept of dtypes as data types, and if no arguments are specified, date_time will take on an object dtype.
End of explanation
nrg['date_time'] = pd.to_datetime(nrg['date_time'])
# https://stackoverflow.com/questions/29206612/difference-between-data-type-datetime64ns-and-m8ns
nrg['date_time'].dtype
Explanation: This will be an issue with any column that can't neatly fit into a single data type.
Working with dates as strings is also an inefficient use of memory and programmer time (not to mention patience).
This exercise will work with time series data, and the date_time column will be formatted as an array of datetime objects called a pandas.Timestamp.
End of explanation
nrg.head()
Explanation: If you're curious about alternatives to the code above, check out pandas.PeriodIndex, which can store ordinal values indicating regular time periods.
We now have a pandas.DataFrame called nrg that contains the data from our .csv file.
Notice how the time is displayed differently in the date_time column.
End of explanation
from timer import timeit
@timeit(repeat=3, number=10)
def convert_with_format(nrg, column_name):
    return pd.to_datetime(nrg[column_name], format='%d/%m/%y %H:%M')
nrg['date_time'] = convert_with_format(nrg, 'date_time')
Explanation: Time for a timing decorator
The code above is pretty straightforward, but how fast does it run?
Let's find out by using a timing decorator called @timeit (an homage to Python's timeit).
This decorator behaves like timeit.repeat(), but it also allows you to return the result of the function itself as well as get the average runtime from multiple trials.
When you create a function and place the @timeit decorator above it, the function will be timed every time it is called.
Keep in mind that the decorator runs an outer and an inner loop.
End of explanation
nrg['cost_cents'] = nrg['energy_kwh'] * 28; nrg.head()
Explanation: One easily overlooked detail is that the datetimes in the energy_consumption.csv file are not in ISO 8601 format.
You need YYYY-MM-DD HH:MM.
If you don’t specify a format, Pandas will use the dateutil package to convert each string to a date.
Conversely, if the raw datetime data is already in ISO 8601 format, pandas can immediately take a fast route to parsing the dates.
This is one reason why being explicit about the format is so beneficial here.
Another option is to pass the infer_datetime_format=True parameter, but it generally pays to be explicit.
Also, remember that pandas' read_csv() method allows you to parse dates as part of the file I/O using the parse_dates, infer_datetime_format, and date_parser parameters.
Simple Looping Over Pandas Data
Now that dates and times are in a tidy format, we can begin calculating electricity costs.
Cost varies by hour, so a cost factor is conditionally applied for each hour of the day: 
|    Usage    | Cents per kWh  |   Time Range   |
|-------------|----------------|----------------|
| Peak        | 28             | 17:00 to 24:00 |
| Shoulder    | 20             | 7:00 to 17:00  |
| Off-Peak    | 12             | 0:00 to 7:00   |
If costs were a flat rate of 28 cents per kilowatt hour every hour, we could just do this:
End of explanation
# Create a function to apply the appropriate rate to the given hour:
def apply_rate(kwh, hour):
    
    Calculates the cost of electricity for a given hour.
    
    if 0 <= hour < 7:
        rate = 12
    elif 7 <= hour <= 17:
        rate = 20
    elif 17 <= hour <= 24:
        rate = 28
    else:
        # +1 for error handling:
        raise ValueError(f'Invalid datetime entry: {hour}')
    return rate * kwh
Explanation: However, our hourly costs depend on the time of day.
If you use a loop to do the conditional calculation, you are not using pandas the way it was intended.
For the rest of this tutorial, you'll start with a sub-optimal solution and work your way up to a Pythonic approach that leverages the full power of pandas.
Take a look at a loop approach and see how it performs using our timing decorator.
End of explanation
# Not the best way:
@timeit(repeat=2, number = 10)
def apply_rate_loop(nrg):
    
    Calculate the costs using a loop, and modify `nrg` dataframe in place.
    
    energy_cost_list = []
    for i in range(len(nrg)):
        # Get electricity used and the corresponding rate.
        energy_used = nrg.iloc[i]['energy_kwh']
        hour = nrg.iloc[i]['date_time'].hour
        energy_cost = apply_rate(energy_used, hour)
        energy_cost_list.append(energy_cost)
    nrg['cost_cents'] = energy_cost_list
apply_rate_loop(nrg)
Explanation: Now for a computationally expensive and non-Pythonic loop:
End of explanation
@timeit(repeat=2, number=10)
def apply_rate_iterrows(nrg):
    energy_cost_list = []
    for index, row in nrg.iterrows():
        energy_used = row['energy_kwh']
        hour = row['date_time'].hour
        energy_cost = apply_rate(energy_used, hour)
        energy_cost_list.append(energy_cost)
    nrg['cost_cents'] = energy_cost_list
apply_rate_iterrows(nrg)
Explanation: You can consider the above to be an “antipattern” in pandas for several reasons.
First, initialize a list in which the outputs will be recorded.
Second, use the opaque object range(0, len(df)) to loop through nrg, then apply apply_rate(), and append the result to a list used to make the new DataFrame column.
Third, chained indexing with df.iloc[i]['date_time'] may lead to unintended results.
Each of these increase the time cost of the calculations.
On my machine, this loop took about 3 seconds for 8760 rows of data.
Next, you’ll look at some improved solutions for iteration over Pandas structures.
Looping with .itertuples() and .iterrows()
Instead of looping through a range of objects, you can use generator methods that yield one row at a time.
.itertuples() yields a namedtuple() for each row, with the row’s index value as the first element of the tuple.
A namedtuple() is a data structure from Python’s collections module that behaves like a Python tuple but has fields accessible by attribute lookup.
.iterrows() yields pairs (tuples) of (index, Series) for each row in the DataFrame.
While .itertuples() tends to be a bit faster, let’s focus on pandas and use .iterrows() in this example.
End of explanation
<END_TASK>