<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Let's start from what we have seen in the previous notebook...
Let's read some wind data
Step1:  Basic information (in this case from a  DataFrame)
Step2:  Access the índexes, values, columns (Series doesn't have this attribute)
Step3:  Removing/extracting columns
There are some columns that are not interesting (named 'x1', 'x2', 'x3', 'x4' and 'x5'). We will remove these columns from our DataFrame. We have several options to do so
Step4:  One of the columns, that extracted using the pop method, is referenced using the  s variable, and it is a Series
Step5:  Is it a TimeSeries?, i.e., Are all the indexes dates?
Step6:  Working with the indexes
Step7:  We can re-write the indexes at any time
Step8:  We can use a column to define our indexes
Step9:  We can undo the set_index action using
Step10:  As with indexes, we can change the name of the columns
Step11:  The indexes 'column' can have a name (already seen before)
Step12:  pandas data structures are numpy arrays on steroids
Don't forget that behind the scenes we have numpy arrays and pandas exposes much of the numpy arrays functionality directly from their data structures.
We can see, for instance, what attributes of a numpy array have an equivalent directly in a Series (or DataFrame)
Step13:  So, a lot of operations we do with a numpy array can be made directly from a pandas data structure
Step14:  ...
<div class="alert alert-danger">
<p><b>Note
Step15:  And where are the steroids?
Be patient!!!!!!
'Stuff' that are in a Series but not in a numpy array
Step16:  'Stuff' that are in a DataFrame but not in a numpy array
Step17:  Examples of some useful operations.
We will see some of this in a more detailed manner and with examples in the next notebooks.
Step18:  Now, we are skimming this. Later we will see it in a more detailed way
Step19:  Other interesting methods are the pd.rolling_*
Step20:  As you can read in the previous warning message the rolling_* functions are deprecated and will not be available in the near future. In the previous text cell I wrote explicitly 'methods' because all the rolling_* functions now are grouped in the rolling method. How we can do it with the rolling method
Step21:  Other interesting 'stuff' in a DataFrame (change DataFrame with Series or other data structures)
Step22:  Working with missing data
It is quite usual that our datasets have missing data.
Step23:  As opposed to what happens with a numpy array, in pandas, operations ignore NaN values unless we explicitly state the opposite. Let's see this in action
Step24:  We can detect 'null' values (NaN) using isnull
Step25:  Or not null using notnull
Step26:  We can see that we have NaN values. We can fill them using ffill or bfill (similar to fillna(method = 'ffill') and to fillna(method = 'bfill'), respectively)
Step27:  Let's create a new DataFrame with indexes with 12H frequency.
Step28:  We can remove rows or columns that have a NaN value, all NaN values,...
Step29:  We can also fill NaN values using interpolate
Step30:  But, what is happening here!!! Why null values are not being interpolated?
Let's see how are the columns.
Step31:  We can see columns 0, 1 y 2 are of type object and this type is not a number. On the other hand, in the column col4 there isn't any value to interpolate. Last, in column col5 all the values are NaN. Let's convert the first three columns to interpolate
<ASSISTANT_TASK:>
Python Code:
# first, the imports
import os
import datetime as dt
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(19760812)
%matplotlib inline
# we read data from file 'mast.txt'
ipath = os.path.join('Datos', 'mast.txt')
# Now, we define a function to parse the dates
def dateparse(date, time):
    YY = 2000 + int(date[:2])
    MM = int(date[2:4])
    DD = int(date[4:])
    hh = int(time[:2])
    mm = int(time[2:])
    
    return dt.datetime(YY, MM, DD, hh, mm, 0)
    
cols = ['Date', 'time', 'wspd', 'wspd_max', 'wdir',
        'x1', 'x2', 'x3', 'x4', 'x5', 
        'wspd_std']
wind = pd.read_csv(ipath, sep = "\s*", names = cols, 
                   parse_dates = [[0, 1]], index_col = 0,
                   date_parser = dateparse)
Explanation: Let's start from what we have seen in the previous notebook...
Let's read some wind data
End of explanation
wind.info()
wind.describe()
Explanation: Basic information (in this case from a  DataFrame)
End of explanation
wind.index
wind.values
wind.values.shape
wind.columns
Explanation: Access the índexes, values, columns (Series doesn't have this attribute):
End of explanation
# We remove a column using 'del' keyword
del wind['x1']
wind.head(3)
# We extract a column using 'pop' method
s = wind.pop('x2')
wind.head(3)
del wind['x3']
del wind['x4']
del wind['x5']
wind.info()
Explanation: Removing/extracting columns
There are some columns that are not interesting (named 'x1', 'x2', 'x3', 'x4' and 'x5'). We will remove these columns from our DataFrame. We have several options to do so:
End of explanation
type(s)
s.head(3)
s.info()
Explanation: One of the columns, that extracted using the pop method, is referenced using the  s variable, and it is a Series:
End of explanation
# s.is_time_series deprecated
s.index.is_all_dates
s.describe()
s.dtype
s.values
s.index
s.columns
Explanation: Is it a TimeSeries?, i.e., Are all the indexes dates?
End of explanation
# We create a DataFrame
df = pd.DataFrame(np.array([['a','b','c','d','e'], [10,20,30,40,50]]).T,
                  columns = ['col1', 'col2'])
df
Explanation: Working with the indexes
End of explanation
df.index = np.arange(1,6) * 100
df
Explanation: We can re-write the indexes at any time:
End of explanation
df.set_index('col1', inplace = True)
df
Explanation: We can use a column to define our indexes:
End of explanation
df.reset_index(inplace = True)
df
Explanation: We can undo the set_index action using:
End of explanation
df.columns = ['column1', 'column2']
df
Explanation: As with indexes, we can change the name of the columns:
End of explanation
df.index.name = 'indices'
df
Explanation: The indexes 'column' can have a name (already seen before):
End of explanation
numpy_attrs = dir(s.values)
series_attrs = dir(s)
for attr in numpy_attrs:
    if attr not in series_attrs:
        print('NOOOOOOOOOOOOOOOOOOOOOO', attr)
    else:
        print(attr)
Explanation: pandas data structures are numpy arrays on steroids
Don't forget that behind the scenes we have numpy arrays and pandas exposes much of the numpy arrays functionality directly from their data structures.
We can see, for instance, what attributes of a numpy array have an equivalent directly in a Series (or DataFrame):
End of explanation
s.mean()
s.min()
s.max()
s[0:10].tolist()
Explanation: So, a lot of operations we do with a numpy array can be made directly from a pandas data structure:
End of explanation
%%timeit 
s.mean()
s.min()
s.max()
%%timeit 
s.values.mean()
s.values.min()
s.values.max()
Explanation: ...
<div class="alert alert-danger">
<p><b>Note:</b></p>
<p>Sometimes could be convenient to use directly the numpy arrays method when performance is an issue.</p>
</div>
End of explanation
numpy_attrs = dir(s.values)
series_attrs = dir(s)
for attr in series_attrs:
    if attr not in numpy_attrs:
        print('NOOOOOOOOOOOOOOOOOOOOOO', attr)
    else:
        print(attr)
Explanation: And where are the steroids?
Be patient!!!!!!
'Stuff' that are in a Series but not in a numpy array
End of explanation
numpy_attrs = dir(s.values)
dataframe_attrs = dir(wind)
for attr in dataframe_attrs:
    if attr not in numpy_attrs:
        print('NOOOOOOOOOOOOOOOOOOOOOO', attr)
    else:
        print(attr)
Explanation: 'Stuff' that are in a DataFrame but not in a numpy array
End of explanation
wind['wspd'].apply(lambda x: str(x) + ' m/s')
wind.corr()
wind.cumsum()
wind.diff()
Explanation: Examples of some useful operations.
We will see some of this in a more detailed manner and with examples in the next notebooks.
End of explanation
# Calculate the mean wind speed (column 'wspd'):
# Calculate the median of the wind direction (column 'wdir'):
# Obtain the maximum difference between two time steps
# (column 'wspd_std')
Explanation: Now, we are skimming this. Later we will see it in a more detailed way:
Let's do some simple examples:
End of explanation
pd.rolling_mean(wind, 5, center = True).head(10)
Explanation: Other interesting methods are the pd.rolling_*:
End of explanation
wind.rolling(5, center = True).mean().head(10)
Explanation: As you can read in the previous warning message the rolling_* functions are deprecated and will not be available in the near future. In the previous text cell I wrote explicitly 'methods' because all the rolling_* functions now are grouped in the rolling method. How we can do it with the rolling method:
End of explanation
import inspect
info = inspect.getmembers(wind, predicate=inspect.ismethod)
for stuff in info:
    print(stuff[0])
Explanation: Other interesting 'stuff' in a DataFrame (change DataFrame with Series or other data structures):
End of explanation
index = pd.date_range('2000/01/01', freq = '12H', periods = 10)
index = index.append(pd.date_range('2000/01/10', freq = '1D', periods = 3))
df = pd.DataFrame(np.random.randint(1, 100, size = (13, 3)), 
                  index = index, columns = ['col1', 'col2', 'col3'])
df
# Let's fill some values with NaN
df[df > 70] = np.nan
df
Explanation: Working with missing data
It is quite usual that our datasets have missing data.
End of explanation
df['col1'].sum()
df['col1'].values.sum()
df['col1'].sum(skipna = False)
Explanation: As opposed to what happens with a numpy array, in pandas, operations ignore NaN values unless we explicitly state the opposite. Let's see this in action:
End of explanation
df.isnull()
Explanation: We can detect 'null' values (NaN) using isnull:
End of explanation
df.notnull()
Explanation: Or not null using notnull:
End of explanation
# Let's remember how is our DataFrame
df
df.ffill()
df.bfill()
df.fillna(value = 'Kiko')
Explanation: We can see that we have NaN values. We can fill them using ffill or bfill (similar to fillna(method = 'ffill') and to fillna(method = 'bfill'), respectively):
End of explanation
df = pd.DataFrame(np.random.randint(1, 100, size = (15, 3)), 
                  index = pd.date_range('2015/01/01', freq = '12H', periods = 15))
df
df[df > 70] = 'Kiko'
df
Explanation: Let's create a new DataFrame with indexes with 12H frequency.
End of explanation
df[df == 'Kiko'] = np.nan
df
# We remove the rows where any value of the row is NaN
# axis = 0 would be equivalent to axis = 'rows' or axis = 'index'
# Later we will see more about the axis keyword...
df.dropna(axis = 'rows') 
# Let's remove the rows where all the values in the row are NaN
df.iloc[2, :] = np.nan
df.dropna(axis = 'rows', how = 'all')
# We can remove columns where any valu in the column is a NaN
df.dropna(axis = 'columns', how = 'any') # axis = 1 is equivalent to axis = 'columns'. More on this later.
                                         # how = 'any' is he default value so we don't need to add it.
# Let's add a column only with not null values and let's repeat the operation
df['col4'] = 9999
df.dropna(axis = 'columns', how = 'any')
# Now let's add a column where all the values are NaN
df['col5'] = np.nan
df.dropna(axis = 'columns', how = 'all')
Explanation: We can remove rows or columns that have a NaN value, all NaN values,...
End of explanation
df.interpolate()
Explanation: We can also fill NaN values using interpolate:
End of explanation
df.info()
Explanation: But, what is happening here!!! Why null values are not being interpolated?
Let's see how are the columns.
End of explanation
df[[0, 1, 2]] = df[[0, 1, 2]].astype(np.float)
df.interpolate()
# Have a look to the docs of the 'interpolate' method to know how to use it
Explanation: We can see columns 0, 1 y 2 are of type object and this type is not a number. On the other hand, in the column col4 there isn't any value to interpolate. Last, in column col5 all the values are NaN. Let's convert the first three columns to interpolate:
End of explanation
<END_TASK>