<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 for some reason Janet's virtualenv is much happier with this TkAgg thing set.
Step1:  make sure none of the phyla are NA (checking 160304 update to load_data.py
Step2:  Test filter and reduce functions using a high threshold, which selects for genus==Methylobacter
Step3:  Demo of DMD data prep
Step4:  Errors are thrown by functions below if you drop min_abunance below.  I think it is hanging up on multiple "other" rows.
Step5:  We can get each dataframe out like this
Step6:  DMD
TODO
Step7:  I'm stuck at the installation of modred 
Step8:  Feature extraction and PCA
Step9:  Just do PCA on a tiny bit of the data as a demo
Step10:  Do correlations for a tiny subset of the data.
<ASSISTANT_TASK:>
Python Code:
import matplotlib as mpl
mpl.use('TkAgg')
import matplotlib.pyplot as plt
%matplotlib inline
import bacteriopop_utils
import feature_selection_utils
import load_data
loaded_data = data = load_data.load_data()
loaded_data.shape
Explanation: for some reason Janet's virtualenv is much happier with this TkAgg thing set.
End of explanation
loaded_data[loaded_data['phylum'].isnull()].head(3)
loaded_data.head()
Explanation: make sure none of the phyla are NA (checking 160304 update to load_data.py
End of explanation
bacteriopop_utils.filter_by_abundance(dataframe=loaded_data, low= 0.6).head()
bacteriopop_utils.reduce_data(dataframe=loaded_data, min_abundance= 0.6, 
                              phylo_column='genus', oxygen='high').head()
Explanation: Test filter and reduce functions using a high threshold, which selects for genus==Methylobacter
End of explanation
raw_dmd_data = bacteriopop_utils.reduce_data(
    dataframe=loaded_data, min_abundance= 0.01, 
    phylo_column='genus', oxygen='Low')
Explanation: Demo of DMD data prep
End of explanation
data_dict = bacteriopop_utils.break_apart_experiments(raw_dmd_data)
data_dict.keys()
# Can't view generators very easily!!!
data_dict.itervalues()
# But we can make a list from them and grab the 0th item
first_df = list(data_dict.itervalues())[0]
first_df.head(3)
first_df[first_df['genus'] == 'other'].head()
first_df[first_df['genus'] != ''].pivot(index='genus', columns='week', values='abundance')
raw_dmd_data.columns
DMD_input_dict = \
    bacteriopop_utils.prepare_DMD_matrices(raw_dmd_data,
                                           groupby_level = "genus")
type(DMD_input_dict)
Explanation: Errors are thrown by functions below if you drop min_abunance below.  I think it is hanging up on multiple "other" rows.
End of explanation
DMD_input_dict[('Low', 1)]
DMD_input_dict[('Low', 1)].shape
DMD_input_dict[('Low', 1)].groupby('week')['abundance'].sum()
Explanation: We can get each dataframe out like this:
End of explanation
DMD_test_matrix = DMD_input_dict[('Low', 1)]
# Who is in there? 
DMD_test_matrix.reset_index()['genus'].unique()
Explanation: DMD
TODO: test DMD on this abundance marix.
End of explanation
# following example 1: https://pythonhosted.org/modred/tutorial_modaldecomp.html
import modred as MR
num_modes = 1
modes, eig_vals = MR.compute_POD_matrices_snaps_method(DMD_test_matrix, range(num_modes))
modes
eig_vals
Explanation: I'm stuck at the installation of modred :(
End of explanation
extracted_features = bacteriopop_utils.extract_features(
    dataframe = loaded_data,
    column_list = ['kingdom', 'phylum', 'class', 'order', 'family', 'genus', 'oxygen', 'abundance']
    # default list was: ['kingdom', 'phylum', 'class', 'order', 'family', 'genus', 'length', 'abundance', 'project']
    )
extracted_features.head()
extracted_features.shape
Explanation: Feature extraction and PCA
End of explanation
pca_results = feature_selection_utils.pca_bacteria(
    data = extracted_features.head(100), n_components = 10)
pca_results.components_
Explanation: Just do PCA on a tiny bit of the data as a demo
End of explanation
feature_selection_utils.calculate_features_target_correlation(
    data = extracted_features.head(100),
    features = extracted_features.columns.tolist(),
    target='abundance',
    method="Pearson")
Explanation: Do correlations for a tiny subset of the data.
End of explanation
<END_TASK>