<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Table of Contents
Hardware
Disk space
RAM memory
CPUs
Software
GEM Mapper
Conda
Python dependencies
DSRC FASTQ compressor
TADbit
Hardware
Disk space
The processing of a "typical" Hi-C experiment of about 200 M reads will occupy a space of around 100 GB per experiment. After the anaysis many of the intermediate files can be compressed or erased, but at it is probable that at each of the experiment/replicate will at least 50 Gb in disk.
RAM memory
The more the better. RAM is specially important to load matrices at high resolution, but usually 32 Gb of RAM should be enough to deal with 50 kb resolution matrices on a human genome.
CPUs
No limitations here, just time. A 8 core computer should be abble to process a single Hi-C experiment (200 M reads, analyzed at 50 kb) in 3-4 days. This includes all the steps of the mapping, filtering, normalization and detection of TADs and compartments.
The 3D modeling will depend on the size of the regions to be modeled.
Software
GEM Mapper
In this course we will use GEM, but any other alternative is just fine.
To install GEM, go to the download page
Step1:  Uncompress the archive
Step2:  And copy the needed binaries to somewhere in your PATH, like
Step3:  In case you do not have root access, just copy the binaries to some path and add this path to your global PATH
Step4:  Conda
Conda (http
Step5:  And run it with all the default options. The installer will create a miniconda2 folder in your home directory where all the programs that you need will be stored (including python).
Python dependencies
With conda you can install the needed dependencies
Step6:  DSRC FASTQ compressor
DSRC is a FASTQ compressor, it's not needed, but we use it as the size of the files is significantly smaller than using gunzip (>30%), and, more importantly, the access to them can be parallelized, and is much faster than any other alternative.
It can be downloaded from https
Step7:  And, if you have root access
Step8:  Otherwise, and as before
Step9:  TADbit
For now TADbit is not available through conda or pip package manager, so to install it we will have to clone the repository, and compile the binaries
<ASSISTANT_TASK:>
Python Code:
! wget -O GEM.tbz2 https://sourceforge.net/projects/gemlibrary/files/gem-library/Binary%20pre-release%202/GEM-binaries-Linux-x86_64-core_i3-20121106-022124.tbz2/download
Explanation: Table of Contents
Hardware
Disk space
RAM memory
CPUs
Software
GEM Mapper
Conda
Python dependencies
DSRC FASTQ compressor
TADbit
Hardware
Disk space
The processing of a "typical" Hi-C experiment of about 200 M reads will occupy a space of around 100 GB per experiment. After the anaysis many of the intermediate files can be compressed or erased, but at it is probable that at each of the experiment/replicate will at least 50 Gb in disk.
RAM memory
The more the better. RAM is specially important to load matrices at high resolution, but usually 32 Gb of RAM should be enough to deal with 50 kb resolution matrices on a human genome.
CPUs
No limitations here, just time. A 8 core computer should be abble to process a single Hi-C experiment (200 M reads, analyzed at 50 kb) in 3-4 days. This includes all the steps of the mapping, filtering, normalization and detection of TADs and compartments.
The 3D modeling will depend on the size of the regions to be modeled.
Software
GEM Mapper
In this course we will use GEM, but any other alternative is just fine.
To install GEM, go to the download page: https://sourceforge.net/projects/gemlibrary/files/gem-library/Binary%20pre-release%202/
and download the i3 version (the other version is for older computers, and you usually won't have to use it).
End of explanation
! tar -xjvf GEM.tbz2
Explanation: Uncompress the archive:
End of explanation
! sudo cp GEM-binaries-Linux-x86_64-core_i3-20121106-022124/gem-mapper /usr/local/bin/
! sudo cp GEM-binaries-Linux-x86_64-core_i3-20121106-022124/gem-indexer /usr/local/bin/
Explanation: And copy the needed binaries to somewhere in your PATH, like:
End of explanation
! mkdir ~/bin
! cp GEM-binaries-Linux-x86_64-core_i3-20121106-022124/gem-mapper ~/bin/
! cp GEM-binaries-Linux-x86_64-core_i3-20121106-022124/gem-indexer* ~/bin/
! echo $PATH=$PATH:"~/bin/" >> ~/.bashrc
Explanation: In case you do not have root access, just copy the binaries to some path and add this path to your global PATH:
End of explanation
! wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh
Explanation: Conda
Conda (http://conda.pydata.org/docs/index.html) is a package manager, mainly hosting python programs, that is very usefull when no root access is available and the softwares have complicated dependencies.
To install is just download the installer from http://conda.pydata.org/miniconda.html
End of explanation
! conda install -y scipy # scientific computing in python
! conda install -y numpy # scientific computing in python
! conda install -y matplotlib # to produce plots
! conda install -y jupyter # this notebook :)
! conda install -y -c https://conda.anaconda.org/bcbio pysam # to deal with SAM/BAM files
! conda install -y -c https://conda.anaconda.org/salilab imp # for 3D modeling
! conda install -y pip # yet another python package manager
! conda install -y -c bioconda mcl # for clustering
Explanation: And run it with all the default options. The installer will create a miniconda2 folder in your home directory where all the programs that you need will be stored (including python).
Python dependencies
With conda you can install the needed dependencies:
End of explanation
! wget http://sun.aei.polsl.pl/dsrc/download/2.0rc/dsrc
! chmod +x dsrc
Explanation: DSRC FASTQ compressor
DSRC is a FASTQ compressor, it's not needed, but we use it as the size of the files is significantly smaller than using gunzip (>30%), and, more importantly, the access to them can be parallelized, and is much faster than any other alternative.
It can be downloaded from https://github.com/lrog/dsrc
End of explanation
sudo mv dsrc /usr/local/bin
Explanation: And, if you have root access:
End of explanation
mv dsrc ~/bin
Explanation: Otherwise, and as before:
End of explanation
! git clone git@github.com:3DGenomes/TADbit.git
! cd TADbit; python setup.py install
Explanation: TADbit
For now TADbit is not available through conda or pip package manager, so to install it we will have to clone the repository, and compile the binaries:
End of explanation
<END_TASK>