<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Data Manipulation with Pandas
Step1:  Categorical Types
Pandas provides a convenient dtype for reprsenting categorical, or factor, data
Step2:  By default the Categorical type represents an unordered categorical
You can provide information about the order of categories
Step3:  Support in DataFrames
When a Categorical is in a DataFrame, there is a special cat accessor
This gives access to all of the features of the Categorical type
Step4:  Exercise
Load NFS data again. Convert fditemno to a Categorical Type. Use describe.
Step5:  Date and Time Types
Pandas provides conveniences for working with dates
Creating a Range of Dates
Step6:  Support in DataFrames
When a datetime type is in a DataFrame, there is a special dt accessor
This gives access to all of the features of the datetime type
Step7:  Indexing with Dates
You can use strings
Note
Step8:  DatetimeIndex supports partial string indexing
Step9:  You can resample to a lower frequency, specifying how to aggregate
Uses the DateTeimIndexResampler object
Step10:  Or go to a higher frequency, optionally specifying how to fill in the
Step11:  There are convenience methods to lag and lead time series
Step12:  Rolling and Window Functions
Pandas also provides a number of convenience functions for working on rolling or moving windows of time series through a common interface
This interface is the new Rolling object
Step13:  Exercise
Create a datetime colume named 'date' for NFS_1974.csv NFS diary data
styr
Step14:  Merging and Joining DataFrames
Step15:  A lot of the time data that comes from relational databases will be normalized
I.e., redundant information will be put in separate tables
Users are expected to merge or join tables to work with them
Step16:  Several ships in the vessels data have traveled multiple segments as we would expect
Matching the names in the transit data to the vessels data is thus a many-to-one match
aside pandas Indices (of which Columns are one) are set-like
Step17:  Merging
We can combine these two datasets for a many-to-one match
merge will use the common columns if we do not explicitly specify the columns
Step18:  Watch out, when merging on columns, indices are discarded
Step19:  Joins
Join is like merge, but it works on the indices
The same could be achieved with merge and the left_index and right_index keywords
Step20:  Exercise
Join the 1974 Household NFS data with the Diary data
The data is in ../data/NationalFoodSurvey/NFS_1974/
Step21:  Concatenation
Another common operation is appending data row-wise or column-wise to an existing dataset
We can use the concat function for this
Let's import two microbiome datasets, each consisting of counts of microorganisms from a particular patient. 
We will use the first column of each dataset as the index.
The index is the unique biological classification of each organism, beginning with domain, phylum, class, and for some organisms, going all the way down to the genus level.
Step22:  We can concatenate on the rows
Step23:  Exercise
Join all of the diary data together in a single DataFrame
Hint
Step24:  Text Data Manipulation
Much like the cat and dt accessors we've already seen
String types have a str accessor that provides fast string operations on columns
Step25:  Count the vessel separators
Step26:  Split on these accessors and expand to return a DataFrame with nan-padding
Step27:  Exercise
Load the file "Ref_ food groups.txt"
Get all of the food groups that contain the word milk
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
pd.set_option('max_rows', 10)
Explanation: Data Manipulation with Pandas
End of explanation
c = pd.Categorical(['a', 'b', 'b', 'c', 'a', 'b', 'a', 'a', 'a', 'a'])
c
c.describe()
c.codes
c.categories
Explanation: Categorical Types
Pandas provides a convenient dtype for reprsenting categorical, or factor, data
End of explanation
c.as_ordered()
Explanation: By default the Categorical type represents an unordered categorical
You can provide information about the order of categories
End of explanation
dta = pd.DataFrame.from_dict({'factor': c,
                              'x': np.random.randn(10)})
dta.head()
dta.dtypes
dta.factor.cat
dta.factor.cat.categories
dta.factor.describe()
Explanation: Support in DataFrames
When a Categorical is in a DataFrame, there is a special cat accessor
This gives access to all of the features of the Categorical type
End of explanation
# [Solution Here]
%load solutions/load_nfs_categorical.py
Explanation: Exercise
Load NFS data again. Convert fditemno to a Categorical Type. Use describe.
End of explanation
dates = pd.date_range("1/1/2015", periods=75, freq="D")
dates
y = pd.Series(np.random.randn(75), index=dates)
y.head()
y.reset_index().dtypes
Explanation: Date and Time Types
Pandas provides conveniences for working with dates
Creating a Range of Dates
End of explanation
dta = (y.reset_index(name='t').
       rename(columns={'index': 'y'}))
dta.head()
dta.dtypes
dta.y.dt.freq
dta.y.dt.day
Explanation: Support in DataFrames
When a datetime type is in a DataFrame, there is a special dt accessor
This gives access to all of the features of the datetime type
End of explanation
y.ix["2015-01-01":"2015-01-15"]
Explanation: Indexing with Dates
You can use strings
Note: the ending index is inclusive here. This is different than most of the rest of Python
End of explanation
y["2015-01"]
Explanation: DatetimeIndex supports partial string indexing
End of explanation
resample = y.resample("M")
resample.mean()
Explanation: You can resample to a lower frequency, specifying how to aggregate
Uses the DateTeimIndexResampler object
End of explanation
y.asfreq('H', method='ffill')
Explanation: Or go to a higher frequency, optionally specifying how to fill in the
End of explanation
y
y.shift(1)
y.shift(-1)
Explanation: There are convenience methods to lag and lead time series
End of explanation
ts = pd.Series(np.random.randn(1000), index=pd.date_range('1/1/2000', 
                                                          periods=1000))
ts = ts.cumsum()
rolling = ts.rolling(window=60)
rolling
rolling.mean()
Explanation: Rolling and Window Functions
Pandas also provides a number of convenience functions for working on rolling or moving windows of time series through a common interface
This interface is the new Rolling object
End of explanation
# [Solution here]
%load solutions/load_nfs_datetime.py
Explanation: Exercise
Create a datetime colume named 'date' for NFS_1974.csv NFS diary data
styr: Survey year
stmth: Survey month
logday: Day in the log (assume logdays are actual days)
Hint: You could do this in two ways
Look at the parse_dates keyword of read_csv
Create the date after reading in the DataFrame
End of explanation
# this is a bit slow because of the date parsing
transit = pd.read_csv("../data/AIS/transit_segments.csv", 
                      parse_dates=['st_time', 'end_time'],
                      infer_datetime_format=True)
vessels = pd.read_csv("../data/AIS/vessel_information.csv")
Explanation: Merging and Joining DataFrames
End of explanation
vessels.head()
transit.head()
Explanation: A lot of the time data that comes from relational databases will be normalized
I.e., redundant information will be put in separate tables
Users are expected to merge or join tables to work with them
End of explanation
vessels.columns.intersection(transit.columns)
Explanation: Several ships in the vessels data have traveled multiple segments as we would expect
Matching the names in the transit data to the vessels data is thus a many-to-one match
aside pandas Indices (of which Columns are one) are set-like
End of explanation
transit.merge(vessels).head()
Explanation: Merging
We can combine these two datasets for a many-to-one match
merge will use the common columns if we do not explicitly specify the columns
End of explanation
A = pd.DataFrame(np.random.randn(25, 2), 
                 index=pd.date_range('1/1/2015', periods=25))
A[2] = np.repeat(list('abcde'), 5)
A
B = pd.DataFrame(np.random.randn(5, 2))
B[2] = list('abcde')
B
A.merge(B, on=2)
Explanation: Watch out, when merging on columns, indices are discarded
End of explanation
transit.set_index('mmsi', inplace=True)
vessels.set_index('mmsi', inplace=True)
transit.join(vessels).head()
Explanation: Joins
Join is like merge, but it works on the indices
The same could be achieved with merge and the left_index and right_index keywords
End of explanation
%load solutions/join_nfs.py
Explanation: Exercise
Join the 1974 Household NFS data with the Diary data
The data is in ../data/NationalFoodSurvey/NFS_1974/
End of explanation
df1 = pd.read_csv('../data/ebola/guinea_data/2014-08-04.csv', 
                  index_col=['Date', 'Description'])
df2 = pd.read_csv('../data/ebola/guinea_data/2014-08-26.csv',
                 index_col=['Date', 'Description'])
df1.shape
df2.shape
df1.head()
df2.head()
df1.index.is_unique
df2.index.is_unique
Explanation: Concatenation
Another common operation is appending data row-wise or column-wise to an existing dataset
We can use the concat function for this
Let's import two microbiome datasets, each consisting of counts of microorganisms from a particular patient. 
We will use the first column of each dataset as the index.
The index is the unique biological classification of each organism, beginning with domain, phylum, class, and for some organisms, going all the way down to the genus level.
End of explanation
df = pd.concat((df1, df2), axis=0)
df.shape
Explanation: We can concatenate on the rows
End of explanation
# [Solution here]
%load solutions/concat_nfs.py
Explanation: Exercise
Join all of the diary data together in a single DataFrame
Hint: you might find glob.glob useful
You will need to add a unique field identifying the survey year to each DataFrame
Hint: you might find a regular expression using re.search useful
End of explanation
vessels.type
Explanation: Text Data Manipulation
Much like the cat and dt accessors we've already seen
String types have a str accessor that provides fast string operations on columns
End of explanation
vessels.type.str.count('/').max()
Explanation: Count the vessel separators
End of explanation
vessels.type.str.split('/', expand=True)
Explanation: Split on these accessors and expand to return a DataFrame with nan-padding
End of explanation
# [Solution here]
%load solutions/nfs_dairy.py
Explanation: Exercise
Load the file "Ref_ food groups.txt"
Get all of the food groups that contain the word milk
End of explanation
<END_TASK>