<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Experiment transforms
Somehow similar to the filters API, the transforms are objects that aim to "transform" the experiment object to a new experiment object.
Again, the returned value should always a new experiment, this allows chaining and pipelining of transforms.
Similar to how filters are applied using the filter methods, the transform API is applied using the apply method.
Examples of transforms are listed in this notebook/chapter.
Step1:  The apply function
The apply function can process three different types of parameters
Step2:  The Rarefaction Transform
A rarefaction transform rarefies our experiment to a desired cutoff.
All samples below this cutoff are discarded from the data DataFrame.
Step3:  The rarefaction functions can also take a num_reps argument (default=1), to set the number of randomizations run.
Step4:  Applying a list of Transforms in sequence
Passing in a list which contains transforms objects/classes chains the transforms in sequence on our experiment.
Step5:  Another very basic transform is the number of unique observations in each sample.
This is essentially the "Number of Observed OTUs", or when used for species, can be described as "species richness".
Step6:  Note how when we implement a Transform, we try to follow the convention of samples being in the columns rather than the rows (Take a look at the source code!)
Applying a numpy aggregate function
Passing in a function object (for example, numpy's sum or mean functions) to apply can be though of as equivalent to the following apply operation on the data_df
Step7:  Rarefaction curves are implemented as a transform
Rarefaction curves is a very common transformation for microbiome amplicon studies and is utilized widely in the software package qiime.
In omicexperiment, it is implemented as a transform which takes three parameters as follows
Step8:  For the sake of fun (and science!), you will find that we can very easily, using minimal code, plot this dataframe as an actual rarefaction curves plot.
Step9:  Not a very elegant curve for this example dataset, but you get the point!
Rarefaction Functions
The RarefactionFunction transform allows us to apply a function on a rarefied dataframe.
Let us use this functionality to apply an alpha diversity function to our data.
The scikit-bio diversity package is an excellent resource for alpha diversity functions (http
Step10:  Note how the resulting dataframe is structured.
The index now is a MultiIndex with rarefaction and rep levels.
At each repetition, the function is applied to the rarefaction level chosen.
We can however, aggregate/collapse these repetitions by passing in the agg_rep argument.
Step11:  Rarefaction curves
Rarefaction curves is a very common transformation for microbiome amplicon studies and is utilized widely in the software package qiime.
In omicexperiment, it is implemented as a transform, the RarefactionCurveFunction.
RarefactionCurveFunction all0ws us to pass functions not only to one rarefaction level (like the RarefactionFunction), but to multiple rarefaction levels (i.e. a curve). The RarefactionCurveFunction constructor signature is the same as for RarefactionFunction, but it needs an additional step argument to specify the levels of rarefaction desired between 0 and n.
The RarefactionCurveFunction constructor takes the following four main parameters
Step12:  We can also use this method to calculate alpha diversity measures as a curve, analogous to the way we used them with RarefactionFunction.
Step13:  Distance Matrix transforms
Distance Matrix transforms internally utilize the cdist function from scipy.spatial
The cdist function accepts either a string for the builtin functions already provided in scipy
(see http
Step14:  You can then apply PCoA on the distance matrix. Here we utilize sci-kit bio's implementation.
Step15:  PCoA is also implemented in omicexperiment as a Transform. It is based on the excellent sci-kit bio's implementation, with minor changes.
<ASSISTANT_TASK:>
Python Code:
%load_ext autoreload
%autoreload 2
#Load our data
from omicexperiment.experiment.microbiome import MicrobiomeExperiment
mapping = "example_map.tsv"
biom = "example_fungal.biom"
tax = "blast_tax_assignments.txt"
exp = MicrobiomeExperiment(biom, mapping,tax)
exp.data_df
exp.mapping_df
Explanation: Experiment transforms
Somehow similar to the filters API, the transforms are objects that aim to "transform" the experiment object to a new experiment object.
Again, the returned value should always a new experiment, this allows chaining and pipelining of transforms.
Similar to how filters are applied using the filter methods, the transform API is applied using the apply method.
Examples of transforms are listed in this notebook/chapter.
End of explanation
from omicexperiment.transforms.general import RelativeAbundance
exp.apply(RelativeAbundance) #note how it returns a new experiment
exp.apply(RelativeAbundance).data_df #peak into the transformed data
Explanation: The apply function
The apply function can process three different types of parameters:
1. The first type of parameter is to pass a Transform object (or class).
The second type of parameter is to pass a function (which accepts a pandas DataFrame, intended here to be the experiment's data_df). The reason for the decision to make this type of function accept a dataframe, rather than an OmicExperiment object is to make the apply functionality compatible with various pandas and numpy aggregate functions like sum, mean etc.
The third type of parameter is a list of Transform objects.
The Relative Abundance Transform
End of explanation
from omicexperiment.transforms.general import Rarefaction
exp.apply(Rarefaction(90000)).data_df
#a shorthand on the MicrobiomeExperiment object -- the rarefy method
exp.rarefy(90000).data_df
Explanation: The Rarefaction Transform
A rarefaction transform rarefies our experiment to a desired cutoff.
All samples below this cutoff are discarded from the data DataFrame.
End of explanation
exp.apply(Rarefaction(n=90000, num_reps=10)).data_df
Explanation: The rarefaction functions can also take a num_reps argument (default=1), to set the number of randomizations run.
End of explanation
exp.apply([Rarefaction(90000), RelativeAbundance]).data_df
Explanation: Applying a list of Transforms in sequence
Passing in a list which contains transforms objects/classes chains the transforms in sequence on our experiment.
End of explanation
#%matplotlib inline
from omicexperiment.transforms.general import NumberUniqueObs
exp.apply(NumberUniqueObs).data_df
Explanation: Another very basic transform is the number of unique observations in each sample.
This is essentially the "Number of Observed OTUs", or when used for species, can be described as "species richness".
End of explanation
import numpy as np
exp.apply(np.mean).data_df
#Note that a transpose() is done on the DataFrame in the inside of
#the apply method, such that sample names remain as columns.
Explanation: Note how when we implement a Transform, we try to follow the convention of samples being in the columns rather than the rows (Take a look at the source code!)
Applying a numpy aggregate function
Passing in a function object (for example, numpy's sum or mean functions) to apply can be though of as equivalent to the following apply operation on the data_df:
DataFrame( experiment.data_df.apply(func) )
Also available is the axis parameter to pass to the apply function of the data_df (Default axis=0 i.e. columns):
DataFrame( experiment.data_df.apply(func, axis=axis) )
End of explanation
from omicexperiment.transforms.general import RarefactionCurve
curve_df = exp.apply(RarefactionCurve(10000, 1000, NumberUniqueObs)).data_df
curve_df
Explanation: Rarefaction curves are implemented as a transform
Rarefaction curves is a very common transformation for microbiome amplicon studies and is utilized widely in the software package qiime.
In omicexperiment, it is implemented as a transform which takes three parameters as follows:
1. The rarefaction curve cutoff level. (Any sample with lower count than this cutoff will be dropped.)
2. The 'step': from 0 to the cutoff. For example, a cutoff value of 1000 and a step of 100 will rarefy at levels 0,100,200,300...,900,1000.
3. The Transform to apply: the transform will be applied at each rarefaction level
The RarefactionCurve Transform applies runs the transformation at each level until the cutoff, thus producing data points for the rarefaction curves.
The end result is demonstrated below. In the example below, we run the "Number of Unique Observations Transform" (i.e. observed_otus, or richness if species) on each level of the curve.
End of explanation
%matplotlib inline
import matplotlib.pyplot as plt
for sample in curve_df:
    plt.plot(curve_df.index, curve_df['sample0'])
    plt.plot(curve_df.index, curve_df['sample1'])
Explanation: For the sake of fun (and science!), you will find that we can very easily, using minimal code, plot this dataframe as an actual rarefaction curves plot.
End of explanation
from omicexperiment.transforms.general import RarefactionFunction
from skbio.diversity import alpha
#we pass the function to the func argument
#the axis to apply to is 0 by default (i.e. on columns = samples)
shannon_rf = RarefactionFunction(n=90000, num_reps=10, func=alpha.shannon, axis=0)
shannon = exp.apply(shannon_rf).data_df
shannon
Explanation: Not a very elegant curve for this example dataset, but you get the point!
Rarefaction Functions
The RarefactionFunction transform allows us to apply a function on a rarefied dataframe.
Let us use this functionality to apply an alpha diversity function to our data.
The scikit-bio diversity package is an excellent resource for alpha diversity functions (http://scikit-bio.org/docs/latest/diversity.html).
End of explanation
#this time we pass the np.mean() function function to the agg_rep argument, to aggregate the results
shannon_rf = RarefactionFunction(n=90000, num_reps=10, func=alpha.shannon, axis=0, agg_rep=np.mean)
shannon = exp.apply(shannon_rf).data_df
shannon #now note how the result produces one result row at the intended rarefaction level.
Explanation: Note how the resulting dataframe is structured.
The index now is a MultiIndex with rarefaction and rep levels.
At each repetition, the function is applied to the rarefaction level chosen.
We can however, aggregate/collapse these repetitions by passing in the agg_rep argument.
End of explanation
#the function is called number_unique_obs in the transforms subpackage
from omicexperiment.transforms.observation import number_unique_obs
num_obs_curve = RarefactionCurveFunction(n=90001, num_reps=1, step=30000, func=number_unique_obs, axis=0)
exp.apply(num_obs_curve).data_df
Explanation: Rarefaction curves
Rarefaction curves is a very common transformation for microbiome amplicon studies and is utilized widely in the software package qiime.
In omicexperiment, it is implemented as a transform, the RarefactionCurveFunction.
RarefactionCurveFunction all0ws us to pass functions not only to one rarefaction level (like the RarefactionFunction), but to multiple rarefaction levels (i.e. a curve). The RarefactionCurveFunction constructor signature is the same as for RarefactionFunction, but it needs an additional step argument to specify the levels of rarefaction desired between 0 and n.
The RarefactionCurveFunction constructor takes the following four main parameters:
1. The rarefaction curve cutoff level n. (Any sample with lower count than this cutoff will be dropped.)
2. The 'num_reps': number of randomizations done at each level. 
3. The 'step': from 0 to the cutoff. For example, a cutoff value of 1000 and a step of 100 will rarefy at levels 0,100,200,300...,900,1000.
4. The function to apply: the function to apply on the dataframe at each rarefaction level (as well as the axis) 
The end result is demonstrated below. In the example below, we run the "Number of Unique Observations Transform" (i.e. observed_otus, or richness) on each level of the curve.
End of explanation
from omicexperiment.transforms.general import RarefactionCurveFunction
#construct a curve with a step of 10000 (0 --> n 90000)
#note we add 1 here to include the last level 90000, as the function uses np.arange internally
shannon_curve = RarefactionCurveFunction(n=90001, num_reps=10, step=10000, func=alpha.shannon, axis=0, agg_rep=np.mean)
exp.apply(shannon_curve).data_df
#again if you wish, you could not pass the agg_rep argument to keep the
#results of your repetitions in the resulting dataframe
shannon_curve = RarefactionCurveFunction(n=90001, num_reps=10, step=10000, func=alpha.shannon)
exp.apply(shannon_curve).data_df
Explanation: We can also use this method to calculate alpha diversity measures as a curve, analogous to the way we used them with RarefactionFunction.
End of explanation
from omicexperiment.transforms.general import DistanceMatrix
rarefied_90000 = exp.rarefy(90000, num_reps=10)
rarefied_90000.apply(DistanceMatrix('braycurtis')).data_df
Explanation: Distance Matrix transforms
Distance Matrix transforms internally utilize the cdist function from scipy.spatial
The cdist function accepts either a string for the builtin functions already provided in scipy
(see http://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.spatial.distance.cdist.html)
or it can accept a function (which accepts two arguments - calculating the distance between them).
The DistanceMatrix Transform constructor accepts the metric (string or function), as explained just above.
End of explanation
from skbio.stats.ordination import pcoa
pcoa_results = pcoa(rarefied_90000.apply(DistanceMatrix('braycurtis')).data_df)
pcoa_results
Explanation: You can then apply PCoA on the distance matrix. Here we utilize sci-kit bio's implementation.
End of explanation
from omicexperiment.transforms.ordination import PCoA
bray_curtis_90000 = rarefied_90000.apply(DistanceMatrix('braycurtis'))
pcoa_bray_curtis_90000 = bray_curtis_90000.apply(PCoA)
pcoa_bray_curtis_90000.data_df
#the scikit-bio OrdinationResults object is stored in the experiment's metadata dict under the key 'pcoa'
print(pcoa_bray_curtis_90000.metadata['pcoa'].eigvals)
pcoa_bray_curtis_90000.metadata['pcoa']
%matplotlib inline
from matplotlib import pyplot
#Plot 2nd vs 3rd principal coordinate
pyplot.scatter(pcoa_bray_curtis_90000.data_df.loc['PC2'], pcoa_bray_curtis_90000.data_df.loc['PC3'])
Explanation: PCoA is also implemented in omicexperiment as a Transform. It is based on the excellent sci-kit bio's implementation, with minor changes.
End of explanation
<END_TASK>