<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Again, read some wind data
Step1:  We read a file of wind data
Step2:  We read a second file of simulated environmental data
Step3:  We have data with a time frequency of 10 minutes (wind) vs the second file data (model) with a time frequency of 1 hour.
Parenthetical remark
Step4:  If we don't use the axis keyword explicitly, by default, operations are over rows (axis = 0), i.e., it uses all the elements of a column
Step5:  If we want to obtain the result on each row, i.e., all the elements of all columns in a row, we should add axis = 1
Step6:  Another example
Step7:  I hope you have a clear idea now about how it works.
Merging/combining pandas data structures
What we will see now is not evident and, in some cases, it is convenient to know how relational algebral works to better understand what it is happening.
Combining data using concat
Step8:  
Step9:  concat allows us to 'merge' pandas data structures using rows and columns. 
What we have seen is not clear!!!??? And you didn't asked!!!???
Let's see a simpler example
Step10:  In general, I use this last option with different column names as it is what I want to do...
Concatenating using the append method
We can get something similar to the previous examples using the append method of the data structures
Step11:  In general, this is not what I want to do. What I want to do is a merge with some logics and to do so we could use pd.merge...
Using pd.merge as in a SQL database
Step12:  Imagine we want to merge two DataFrames using columns
Step13:  Combining using the join method
More about the same. The join method helps us to combine pandas data structures. Some quick examples
Step14:  Grouping
We can group information of our data structures in a simple way using the groupby method. In general, here we follow a strategy of split-apply-combine. What we do is, first separate the initial dataset in groups of interest, over each group we apply some calculations and, finally, the results obtained on each group is combined in a new data structure.
Step15:  Let's see what groupby returns
Step16:  pandas.core.groupby.DataFrameGroupBy is like a dict with superpowers!!!
Reshaping/transforming our data structures
<div class="alert alert-info">
<p>Most of this section has been extracted from <a href="https
Step17:  Pivoting using several columns
Step18:  What happens if we combine repeated indexes. For instance
Step19:  (source
Step20:  pivot_table to the rescue to solve the previous error
The previous error can be solved using pivot_table that is more flexible than pivot
<ASSISTANT_TASK:>
Python Code:
# First, the imports
import os
import datetime as dt
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from IPython.display import display
np.random.seed(19760812)
%matplotlib inline
Explanation: Again, read some wind data
End of explanation
# read file 'mast.txt'
ipath = os.path.join('Datos', 'mast.txt')
def dateparse(date, time):
    YY = 2000 + int(date[:2])
    MM = int(date[2:4])
    DD = int(date[4:])
    hh = int(time[:2])
    mm = int(time[2:])
    
    return dt.datetime(YY, MM, DD, hh, mm, 0)
    
cols = ['Date', 'time', 'wspd', 'wspd_max', 'wdir',
        'x1', 'x2', 'x3', 'x4', 'x5', 
        'wspd_std']
wind = pd.read_csv(ipath, sep = "\s*", names = cols, 
                   parse_dates = {'Timestamp': [0, 1]}, index_col = 0,
                   date_parser = dateparse)
Explanation: We read a file of wind data
End of explanation
# read file 'model.txt'
ipath = os.path.join('Datos', 'model.txt')
model = pd.read_csv(ipath, sep = "\s*", skiprows = 3,
                    parse_dates = {'Timestamp': [0, 1]}, index_col = 'Timestamp')
for c in ['x1','x2','x3','x4','x5']: # remove unnecesary columns
    _ = wind.pop(c)
wind.head(3)
model.head(3)
wind['Timestamp'] = wind.index
print(wind['Timestamp'].diff().min())
del wind['Timestamp']
model['Timestamp'] = model.index
print(model['Timestamp'].diff().min())
del model['Timestamp']
Explanation: We read a second file of simulated environmental data
End of explanation
df = pd.DataFrame(np.array([[1, 10], [2, 20], [3,30]]), columns = ['A', 'B'])
df
Explanation: We have data with a time frequency of 10 minutes (wind) vs the second file data (model) with a time frequency of 1 hour.
Parenthetical remark: axis 101
In some occasions we will find a keyword called axis. Let's see in a moment how it works in pandas to try to avoid some issues:
Posibilities
axis = 0 (acts over rows)
axis = 1 (acts over columns)
<span style="color:#888">axis = 2 (only for Panel)</span>
(source: http://stackoverflow.com/a/25774395/5216568).
<br>
<div class="alert alert-info">
<p><b>Flashcard:</b></p> 
<p><a href="https://www.reddit.com/r/pystats/comments/2z0xbc/pandas_axis0_or_axis1_not_intuitive_for_you_use/cpev7x9">Easiest way to remember is that "1" looks like a column!</a></p>
<p><b>Other options:</b></p> 
<p>One option would be to use `axis = 'index'` (similar to `axis = 0`) or `axis = 'columns'` (similar to `axis = 1`) for `DataFrame`s. In the case of `Panel`s we would have `items`, `minor`, `major` (similar to options 0, 1 or 2).</p>.
<p>For a `DataFrame`s you could also use `index = 'rows'`, I think it is more evident than `'index'` but I don't recommend it as it is not documented.</p>
<p>Also, using `'index'`, `'rows'`, `'columns'`,..., can be confuse as in a lot of places we will find keywordscalled like this.</p>.
</div>
But, what is the meaning of 'acts over rows/columns'. Let's see some simple examples to check if it is clear:
End of explanation
df.sum()
# The previous example would be similar to
df.sum(axis = 0)
Explanation: If we don't use the axis keyword explicitly, by default, operations are over rows (axis = 0), i.e., it uses all the elements of a column:
End of explanation
df.sum(axis = 1)
Explanation: If we want to obtain the result on each row, i.e., all the elements of all columns in a row, we should add axis = 1:
End of explanation
df < 10
(df < 10).all()
(df < 10).all(axis = 'columns') # instead of axis = 1 we use axis = 'columns'
# test operations of a DatFrame using axis = 0, 1, 'index', rows', columns'
Explanation: Another example:
End of explanation
new = pd.concat([wind, model], axis = 0, join = 'outer')
new.head(5)
new.tail(5)
new.loc['2014/01/01 00:00':'2014/01/01 02:00']
Explanation: I hope you have a clear idea now about how it works.
Merging/combining pandas data structures
What we will see now is not evident and, in some cases, it is convenient to know how relational algebral works to better understand what it is happening.
Combining data using concat
End of explanation
new = pd.concat([wind, model], axis = 1, join = 'inner')
new.head(5)
new.loc['2014/01/01 00:00':'2014/01/01 02:00']
Explanation: 
End of explanation
df1 = pd.DataFrame(np.random.randn(10,2), 
                   columns = ['A', 'B'], 
                   index = np.arange(10))
df2 = pd.DataFrame(np.random.randn(4,3), 
                   columns = ['A', 'B', 'C'], 
                   index = np.arange(8, 12))
df1
df2
new = pd.concat([df1, df2], axis = 0, join = 'inner')
new
new = pd.concat([df1, df2], axis = 1, join = 'inner')
new
Explanation: concat allows us to 'merge' pandas data structures using rows and columns. 
What we have seen is not clear!!!??? And you didn't asked!!!???
Let's see a simpler example:
End of explanation
wind.append(model)
Explanation: In general, I use this last option with different column names as it is what I want to do...
Concatenating using the append method
We can get something similar to the previous examples using the append method of the data structures:
End of explanation
pd.merge(wind, model, left_index = True, right_index = True, how = 'inner').head(5)
(pd.merge(wind, model, left_index = True, right_index = True, how = 'inner') == 
 pd.concat([wind, model], axis = 1, join = 'inner')).all().all()
Explanation: In general, this is not what I want to do. What I want to do is a merge with some logics and to do so we could use pd.merge...
Using pd.merge as in a SQL database
End of explanation
df1 = pd.DataFrame(
    np.array([
        np.arange(1, 11),
        np.random.choice([1,2,3], size = 10),
        np.arange(1, 11) * 10
    ]).T,
    columns = ['A', 'col', 'B']
)
df2 = pd.DataFrame(
    np.array([
        np.arange(11, 21),
        np.random.choice([1,2,3], size = 10),
        np.arange(1, 11) * 100
    ]).T,
    columns = ['A', 'col', 'B']
)
display(df1)
display(df2)
pd.merge(df1, df2, on = ['col'])
# Play with it with pd.merge keywords to become more comfortable with it
Explanation: Imagine we want to merge two DataFrames using columns:
End of explanation
wind.join(model).head(10)
model.join(wind).head(10)
joinA  = wind.join(model, how = 'inner') 
joinB = model.join(wind, how = 'inner').loc[:,joinA.columns]
(joinA == joinB).all().all()
Explanation: Combining using the join method
More about the same. The join method helps us to combine pandas data structures. Some quick examples:
End of explanation
wind['month'] = wind.index.month
wind.iloc[[0, 1000, 10000, 30000]]
wind.groupby(by = 'month').mean()
wind.groupby(by = [wind.index.year, 'month']).mean()
del wind['month']
# Play grouping 
# (obtain daily mean wind speed, 
# mean wind speed on Tuesdays when wind direction is between 300º and 360º,...)
Explanation: Grouping
We can group information of our data structures in a simple way using the groupby method. In general, here we follow a strategy of split-apply-combine. What we do is, first separate the initial dataset in groups of interest, over each group we apply some calculations and, finally, the results obtained on each group is combined in a new data structure.
End of explanation
grouped = wind.groupby(by=wind.index.month)
import inspect
info = inspect.getmembers(grouped, predicate=inspect.ismethod)
for stuff in info:
    print(stuff[0])
grouped
grouped.ngroups
grouped.groups.keys()
grouped.get_group(2)
Explanation: Let's see what groupby returns
End of explanation
wind['year'] = wind.index.year
wind['month'] = wind.index.month
tmp = wind.groupby(by = ['year', 'month']).mean()
del wind['year']
del wind['month']
tmp
tmp['year'] = tmp.index.get_level_values(0)
tmp['month'] = tmp.index.get_level_values(1)
tmp
tmp.pivot(index = 'year', columns = 'month', values='wspd')
# Get the yearly mean wind speed 
# starting from tmp.pivot(index = 'year', columns = 'month', values='wspd')
Explanation: pandas.core.groupby.DataFrameGroupBy is like a dict with superpowers!!!
Reshaping/transforming our data structures
<div class="alert alert-info">
<p>Most of this section has been extracted from <a href="https://nikolaygrozev.wordpress.com/2015/07/01/reshaping-in-pandas-pivot-pivot-table-stack-and-unstack-explained-with-pictures/">excellent article</a>
<em>Reshaping in Pandas – Pivot, Pivot-Table, Stack and Unstack explained with Pictures</em> by <b>Nikolay Grozev</b>.</p>
<p>Kudos to Nikolay.</p>
<p>Kudos to me because a followed the <a href="https://en.wikipedia.org/wiki/Don't_repeat_yourself">DRY</a> and <a href="https://en.wikipedia.org/wiki/KISS_principle">KISS</a> principles.</p>
</div>
Reshaping allows us to change our data structure in a new one to perform new analyses on the new recombined data.
Pivot
We obtain a new table derived from our initial data table. For instance, imagine I want to obtain monthly mean wind speed on each year.
End of explanation
tmp = wind.groupby(by = [wind.index.year, wind.index.month])
tmp = tmp.agg({'wspd': np.mean, 'wspd_max': 'max'})
tmp.reset_index(inplace = True)
tmp
tmp.pivot(index = 'level_1', columns = 'level_0')
tmp.pivot(index = 'level_1', columns = 'level_0').columns
Explanation: Pivoting using several columns:
End of explanation
from collections import OrderedDict
table = OrderedDict((
    ("Item", ['Item0', 'Item0', 'Item0', 'Item1']),
    ('CType',['Gold', 'Bronze', 'Gold', 'Silver']),
    ('USD',  ['1$', '2$', '3$', '4$']),
    ('EU',   ['1€', '2€', '3€', '4€'])
))
df = pd.DataFrame(table)
df
Explanation: What happens if we combine repeated indexes. For instance:
End of explanation
pivoted = df.pivot(index='Item', columns='CType', values='USD')
Explanation: (source: https://nikolaygrozev.files.wordpress.com/2015/07/pivoting_simple_error.png)
End of explanation
table = OrderedDict((
    ("Item", ['Item0', 'Item0', 'Item0', 'Item1']),
    ('CType',['Gold', 'Bronze', 'Gold', 'Silver']),
    ('USD',  [1, 2, 3, 4]),
    ('EU',   [1.1, 2.2, 3.3, 4.4])
))
df = pd.DataFrame(table)
pivoted = df.pivot_table(index='Item', columns='CType', values='USD', aggfunc=np.min)
pivoted
Explanation: pivot_table to the rescue to solve the previous error
The previous error can be solved using pivot_table that is more flexible than pivot:
End of explanation
<END_TASK>