<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Introduction to Porekit-Python
Disclaimer
Porekit is the result of my personal interest in nanopore sequencing. I'm not affiliated with Oxford Nanopore Technologies, or any MAP participant. This means a lot of the factual information presented in this notebook may be wrong.
General philosophy and goal of Porekit-Python
This library is meant to provide tools for interactively exploring nanopore data inside the Jupyter notebook, and for writing simple scripts or more complex software dealing with nanopore data. Therefore a lot of attention has been given to make interactive use easy and painless, and to keep the code in the background flexible and exposed to library users.
What Oxford Nanopore Data looks like
The MinION sequencer is attached to a laptop running MinKnow. This program connects directly to the MinION device and tells it what to do. Optionally, third party software can connect to an API inside the primary software to remote control the sequencing process. That is not covered here, though.
In a nutshell, nanopore sequencing works by dragging a DNA molecule through a tiny pore in a membrane. As the DNA passes, the voltage difference between the two sides of the membrane change, depending on the electrochemical properties of the passing nucleotides. This means that, at the core of the nanopore data, there is a timeseries of voltage measurements, which is called the "squiggle".
The process to convert the squiggle into a sequence of DNA letters is called base calling. The current MinKnow software uploads the squiggle to Metrichor servers, which perform the base calling, and send the result back to the user's computer.
The result of a sequencing run is a collection of FAST5 files, each containing data on one molecule of DNA which passed through one of currently 512 channels in the flowcell. These files are stored on disk, usually in one directory per run. A convention seems to be to name each file with a unique and descriptive string
Step1:  These files belong to data publishd by Quick et al. http
Step2:  The result is a Pandas DataFrame object, which is too big to comfortably view in its entirety, but still comparatively "small data". Here is a subset of it
Step3:  All of the columns available
Step4:  The philosopy of porekit is to gather the metadata once and then store it in a different format. This makes it easier and faster to analyse the metadata or use it in another context, for example with alignment data. In general, Fast5 Files don't change after MinKNOW and Metrichor, and possibly some third party programs are done with it.
The recommended file format to save this metadata is the "Feather File Format" (https
Step5:  Grouping by Device, ASIC and Run Ids
<ASSISTANT_TASK:>
Python Code:
!ls /home/andi/nanopore/GenomeRU2/downloads/pass/ | tail -n 10
Explanation: Introduction to Porekit-Python
Disclaimer
Porekit is the result of my personal interest in nanopore sequencing. I'm not affiliated with Oxford Nanopore Technologies, or any MAP participant. This means a lot of the factual information presented in this notebook may be wrong.
General philosophy and goal of Porekit-Python
This library is meant to provide tools for interactively exploring nanopore data inside the Jupyter notebook, and for writing simple scripts or more complex software dealing with nanopore data. Therefore a lot of attention has been given to make interactive use easy and painless, and to keep the code in the background flexible and exposed to library users.
What Oxford Nanopore Data looks like
The MinION sequencer is attached to a laptop running MinKnow. This program connects directly to the MinION device and tells it what to do. Optionally, third party software can connect to an API inside the primary software to remote control the sequencing process. That is not covered here, though.
In a nutshell, nanopore sequencing works by dragging a DNA molecule through a tiny pore in a membrane. As the DNA passes, the voltage difference between the two sides of the membrane change, depending on the electrochemical properties of the passing nucleotides. This means that, at the core of the nanopore data, there is a timeseries of voltage measurements, which is called the "squiggle".
The process to convert the squiggle into a sequence of DNA letters is called base calling. The current MinKnow software uploads the squiggle to Metrichor servers, which perform the base calling, and send the result back to the user's computer.
The result of a sequencing run is a collection of FAST5 files, each containing data on one molecule of DNA which passed through one of currently 512 channels in the flowcell. These files are stored on disk, usually in one directory per run. A convention seems to be to name each file with a unique and descriptive string:
End of explanation
import porekit
everything = porekit.gather_metadata("/home/andi/nanopore/", workers=4)
Explanation: These files belong to data publishd by Quick et al. http://europepmc.org/abstract/MED/25386338;jsessionid=ijHIHUVXlcpxeTzVUihz.0
Gathering Metadata
The following snippet will extract meta data from all of my downloaded nanopore data, searching directories recursively.
End of explanation
everything[['asic_id', 'channel_number', 'template_length', 'complement_length']].head()
Explanation: The result is a Pandas DataFrame object, which is too big to comfortably view in its entirety, but still comparatively "small data". Here is a subset of it:
End of explanation
everything.columns
Explanation: All of the columns available:
End of explanation
everything.to_hdf("everything.h5", "meta")
Explanation: The philosopy of porekit is to gather the metadata once and then store it in a different format. This makes it easier and faster to analyse the metadata or use it in another context, for example with alignment data. In general, Fast5 Files don't change after MinKNOW and Metrichor, and possibly some third party programs are done with it.
The recommended file format to save this metadata is the "Feather File Format" (https://github.com/wesm/feather). It is a binary format for DataFrames, and has excellent support for R.
Feather's documentation emphasizes the file format can still change, and thus should not be used for long term storage. It's still fine for most medium term purposes, and even in the longer term, pinning the package versions will make older data available just as easily.
End of explanation
g = everything.groupby(['device_id', 'asic_id', 'run_id'])
df = g.template_length.agg([lambda v: len(v), np.mean, np.max])
df.columns = ['Count', 'Mean template length', 'Max template_length']
df
Explanation: Grouping by Device, ASIC and Run Ids
End of explanation
<END_TASK>