<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Required inputs for Akita are
Step1:  Download a few Micro-C datasets, processed using distiller (https
Step2:  Write out these cooler files and labels to a samples table.
Step3:  Next, we want to choose genomic sequences to form batches for stochastic gradient descent, divide them into training/validation/test sets, and construct TFRecords to provide to downstream programs.
The script akita_data.py implements this procedure.
The most relevant options here are
Step4:  The data for training is now saved in data/1m as tfrecords (for training, validation, and testing), where contigs.bed contains the original large contiguous regions from which training sequences were taken, and sequences.bed contains the train/valid/test sequences.
Step5:  Now train a model!
(Note
<ASSISTANT_TASK:>
Python Code:
import json
import os
import shutil
import subprocess
if not os.path.isfile('./data/hg38.ml.fa'):
    print('downloading hg38.ml.fa')
    subprocess.call('curl -o ./data/hg38.ml.fa.gz https://storage.googleapis.com/basenji_barnyard/hg38.ml.fa.gz', shell=True)
    subprocess.call('gunzip ./data/hg38.ml.fa.gz', shell=True)
Explanation: Required inputs for Akita are:
* binned Hi-C or Micro-C data stored in cooler format (https://github.com/mirnylab/cooler)
* Genome FASTA file
First, make sure you have a FASTA file available consistent with genome used for the coolers. Either add a symlink for a the data directory or download the machine learning friendly simplified version in the next cell.
End of explanation
if not os.path.exists('./data/coolers'):
    os.mkdir('./data/coolers')
if not os.path.isfile('./data/coolers/HFF_hg38_4DNFIP5EUOFX.mapq_30.2048.cool'):
    subprocess.call('curl -o ./data/coolers/HFF_hg38_4DNFIP5EUOFX.mapq_30.2048.cool'+
            ' https://storage.googleapis.com/basenji_hic/tutorials/coolers/HFF_hg38_4DNFIP5EUOFX.mapq_30.2048.cool', shell=True)
    subprocess.call('curl -o ./data/coolers/H1hESC_hg38_4DNFI1O6IL1Q.mapq_30.2048.cool'+
            ' https://storage.googleapis.com/basenji_hic/tutorials/coolers/H1hESC_hg38_4DNFI1O6IL1Q.mapq_30.2048.cool', shell=True)
ls ./data/coolers/
Explanation: Download a few Micro-C datasets, processed using distiller (https://github.com/mirnylab/distiller-nf), binned to 2048bp, and iteratively corrected.
End of explanation
lines = [['index','identifier','file','clip','sum_stat','description']]
lines.append(['0', 'HFF', './data/coolers/HFF_hg38_4DNFIP5EUOFX.mapq_30.2048.cool', '2', 'sum', 'HFF'])
lines.append(['1', 'H1hESC', './data/coolers/H1hESC_hg38_4DNFI1O6IL1Q.mapq_30.2048.cool', '2', 'sum', 'H1hESC'])
samples_out = open('data/microc_cools.txt', 'w')
for line in lines:
    print('\t'.join(line), file=samples_out)
samples_out.close()
Explanation: Write out these cooler files and labels to a samples table.
End of explanation
if os.path.isdir('data/1m'):
    shutil.rmtree('data/1m')
! akita_data.py --sample 0.05 -g ./data/hg38_gaps_binsize2048_numconseq10.bed -l 1048576 --crop 65536 --local -o ./data/1m --as_obsexp -p 8 -t .1 -v .1 -w 2048 --snap 2048 --stride_train 262144 --stride_test 32768 ./data/hg38.ml.fa ./data/microc_cools.txt
Explanation: Next, we want to choose genomic sequences to form batches for stochastic gradient descent, divide them into training/validation/test sets, and construct TFRecords to provide to downstream programs.
The script akita_data.py implements this procedure.
The most relevant options here are:
| Option/Argument | Value | Note |
|:---|:---|:---|
| --sample | 0.1 | Down-sample the genome to 10% to speed things up here. |
| -g | data/hg38_gaps_binsize2048_numconseq10.bed | Dodge large-scale unmappable regions determined from filtered cooler bins. |
| -l | 1048576 | Sequence length. |
| --crop | 65536 | Crop edges of matrix so loss is only computed over the central region. |
| --local | True | Run locally, as opposed to on a SLURM scheduler. |
| -o | data/1m | Output directory |
| -p | 8 | Uses multiple concourrent processes to read/write. |
| -t | .1 | Hold out 10% sequences for testing. |
| -v | .1 | Hold out 10% sequences for validation. |
| -w | 2048 | Pool the nucleotide-resolution values to 2048 bp bins. |
| fasta_file| data/hg38.ml.fa | FASTA file to extract sequences from. |
| targets_file | data/microc_cools.txt | Target table with cooler paths. |
Note: make sure to export BASENJIDIR as outlined in the basenji installation tips 
(https://github.com/calico/basenji/tree/master/#installation).
End of explanation
! cut -f4 data/1m/sequences.bed | sort | uniq -c
! head -n3 data/1m/sequences.bed
Explanation: The data for training is now saved in data/1m as tfrecords (for training, validation, and testing), where contigs.bed contains the original large contiguous regions from which training sequences were taken, and sequences.bed contains the train/valid/test sequences.
End of explanation
# specify model parameters json to have only two targets
params_file   = './params.json'
with open(params_file) as params_file:
    params_tutorial = json.load(params_file)   
params_tutorial['model']['head_hic'][-1]['units'] =2
with open('./data/1m/params_tutorial.json','w') as params_tutorial_file:
    json.dump(params_tutorial,params_tutorial_file) 
    
### note that training with default parameters requires GPU with >12Gb RAM ###
! akita_train.py -k -o ./data/1m/train_out/  ./data/1m/params_tutorial.json ./data/1m/
Explanation: Now train a model!
(Note: for training production-level models, please remove the --sample option when generating tfrecords)
End of explanation
<END_TASK>