bash returning erroneous results after about 36 million lines when iterating through a pair of files - is this a memory error?
<p>I have written a simple script in bash to iterate through a pair of text files to make sure they are properly formatted. </p>

<p>The required format is as follows:</p>

<ul>
<li>Each file contains millions of ‘records’.  </li>
<li>Each record takes up two lines in each file – a header line and a sequence line. </li>
<li>Each header lines consists of a “>” symbol, followd by a sample name (alphanumeric string), followed by a period, followed by a unique record identifier number (an integer), followed by a suffix of either ‘/1’ or ‘/2’.</li>
<li>Each sequence line contains a string of 30-100 A,C,G and T characters (the four DNA nucleotides, if anyone is wondering). </li>
<li>The files are paired, in that the first record in one file corresponds to the first record in the second file,  and so forth.  The header lines in the two files should be identical, except that in one files they will all have a ‘/1’ siffix and in the other file they will all have a ‘/2’ suffix. The sequence lines can be very different between the two files.  </li>
</ul>

<p>The code I developed is designed to check that  (a) the hearder lines in each record follow the correct format, (2) the header lines in the corresponding records in the two files match (i.e. are identical except for the /1 and /2 suffixes) and (c) the sequence lines contain only A,C,G and T characters. </p>

<p>Example of properly formatted records:</p>

<pre><code>&gt; cat -n file1 | head -4 
1  &gt;SRR573705.1/1
2  ATAATCATTTGCCTCTTAAGTGGGGGCTGGTATGAATGGCAAGACGGGAATCTAGCTGTCTCTCCCTTATATCTTGAAGTTAATATTTCTGTGAAGAAGC
3  &gt;SRR573705.2/1
4  CCACTTGTCCCAGTCTGTGCTGCCTGTACAATGGATTAGCTGAGGAAAACTGGCATCCCATGGCCTCAAACAGACGCAGCAAGTCCATGAAGCCATAATT

&gt; cat –n file2 | head -4 
1  &gt;SRR573705.1/2
2  TTTCTAACAATTGAATTAGCAACACAAACACTATTGACAAAGCTATATCTTATTTCTACTAAAGCTCGATAGGGTCTTCTCGTCCTGCGATCCCATTCCT
3  &gt;SRR573705.2/2
4  GTATGATGGGTGTGTCAAGGAGCTCAACCATCGTGATAGGCTACCTCATGCATCGAGACAAGATCACATTTAATGAGGCATTTGACATGGTCAGGAAGCA
</code></pre>

<p>My code is below. It works perfectly well for small test files containing only a couple of hundred records. When reading a real data file with millions or records, however, it returns non-sensical errors, for example:</p>

<pre><code>Inaccurate header line in read 18214236 of file2 
Line 36428471: TGATTTCCTCCATAAGTGCCTTCTCGCACTCAACATCTTGATCACTACGTTCCTCAGCATTCGCCTCTTCTTCTTCTTCCTGTTCCTTTTTTTCATCCTC
</code></pre>

<p>The error above is simply wrong. Line  36,428,471 of file2 is  ‘>SRR573705.19887618/2’</p>

<p>The string reported in the error is not even present in file 2.  It does, however, appear multiple times in file1, i.e.:</p>

<pre><code>cat -n /file1 | grep 'TGATTTCCTCCATAAGTGCCTTCTCGCACTCAACATCTTGATCACTACGTTCCTCAGCATTCGCCTCTTCTTCTTCTTCCTGTTCCTTTTTTTCATCCTC'
4632838 TGATTTCCTCCATAAGTGCCTTCTCGCACTCAACATCTTGATCACTACGTTCCTCAGCATTCGCCTCTTCTTCTTCTTCCTGTTCCTTTTTTTCATCCTC
24639990        TGATTTCCTCCATAAGTGCCTTCTCGCACTCAACATCTTGATCACTACGTTCCTCAGCATTCGCCTCTTCTTCTTCTTCCTGTTCCTTTTTTTCATCCTC
36428472        TGATTTCCTCCATAAGTGCCTTCTCGCACTCAACATCTTGATCACTACGTTCCTCAGCATTCGCCTCTTCTTCTTCTTCCTGTTCCTTTTTTTCATCCTC
143478526       TGATTTCCTCCATAAGTGCCTTCTCGCACTCAACATCTTGATCACTACGTTCCTCAGCATTCGCCTCTTCTTCTTCTTCCTGTTCCTTTTTTTCATCCTC
</code></pre>

<p>The data in the two files seems to match perfectly in the region where the error was returned:</p>

<pre><code>cat -n file1 | head -36428474 | tail
36428465        &gt;SRR573705.19887614/1
36428466        CACCCCAGCATGTTGACCACCCATGCCATTATTTCATGGTATTTTCTTACATTTTGTATATAACAGATGCATTACGTATTATAGCATTGCTTTTCGTAAA
36428467        &gt;SRR573705.19887616/1
36428468        AGATCCTCCTCCTCATCGGTCAGTCGCCAATCCAACAACTCAACCTTCTTCTTCAAGTCACTCAGCCGTCGGCCCGGGACTGCCGTTTCATGATGCCTAT
36428469        &gt;SRR573705.19887617/1
36428470        CAATAGCGTATATTAAAATTGCTGCAGTTAAAAAGCTCGTAGTTGGATCTTGGGCGCAGGCTGGCGGTCCGCCGCAAGGCGCGCCACTGCCAGCCTGGCC
36428471        &gt;SRR573705.19887618/1
36428472        TGATTTCCTCCATAAGTGCCTTCTCGCACTCAACATCTTGATCACTACGTTCCTCAGCATTCGCCTCTTCTTCTTCTTCCTGTTCCTTTTTTTCATCCTC
36428473        &gt;SRR573705.19887619/1
36428474        CCAGCCTGCGCCCAAGATCCAACTACGAGCTTTTTAACTGCAGCAATTTTAATATACGCTATTGGAGCTGGAATTACCGCGGCTGCTGGCACCAGACTTG

&gt;cat -n file2 | head -36428474 | tail
36428465        &gt;SRR573705.19887614/2
36428466        GTAATTTACAGGAATTGTTTACATTCTGAGCAAATAAAACAAATAATTTTAATACACAAACTTGTTGAAAGTTAATTAGGTTTTACGAAAA
36428467        &gt;SRR573705.19887616/2
36428468        GCCGTCGCAGCAACATTTGAGATATCCCGTAAGACGTCTTGAACGGCTGGCTCTGTCTGCTCTCGGAGAACCTGCCGGCTGAACCGGACAGCGCAGACG
36428469        &gt;SRR573705.19887617/2
36428470        CTCGAGTTCCGAAAACCAACGCAATAGAACCGAGGTCCTATTCCATTATTCCATGCTCTGCTGTCCAGGCGGTCGGCCTG
36428471        &gt;SRR573705.19887618/2
36428472        GGACATGGAAACAGAAAATAATGAAAAGACCAAAGAAGATGCACTTGAGGTTGATAAGCCTAAAGG
36428473        &gt;SRR573705.19887619/2
36428474        CCCGACACGGGGAGGTAGTGACGAAAAATAGCAATACAGGACTCTTTCGAGGCCCTGTAATTGGAATGAGTACACTTTAAATCCTTTAACGAGGATCTAT
</code></pre>

<p>Is there some sort of memory limit in bash that could cause such an error? I have run various versions of this code over multiple files and consistently get this problem after 36,000,000 lines.</p>

<p>My code:</p>

<pre><code>set -u    
function fastaConsistencyChecker {
    F_READS=$1
    R_READS=$2
    echo -e $F_READS
    echo -e $R_READS
    if [[ ! -s $F_READS ]]; then echo -e "File $F_READS could not be found."; exit 0; fi
    if [[ ! -s $R_READS ]]; then echo -e "File $R_READS could not be found."; exit 0; fi    
    exec 3&lt;$F_READS
    exec 4&lt;$R_READS        
    line_iterator=1
    read_iterator=1        
    while read FORWARD_LINE &lt;&amp;3 &amp;&amp; read REVERSE_LINE &lt;&amp;4; do
        if [[ $(( $line_iterator % 2 )) == 1 ]]; then
            ## This is a header line ##
            if [[  ! ( $FORWARD_LINE =~ ^"&gt;"[[:alnum:]]+\.[0-9]+/1$ ) ]]; then
                echo -e "Inaccurate header line in read ${read_iterator} of file ${F_READS}"
                echo -e "Line ${line_iterator}: ${FORWARD_LINE}"
                exit 0
            fi

            if [[ ! ( $REVERSE_LINE =~ ^"&gt;"[[:alnum:]]+\.[0-9]+/2$ ) ]]; then
                echo -e "Inaccurate header line in read ${read_iterator} of file ${R_READS}"
                echo -e "Line ${line_iterator}: ${REVERSE_LINE}"
                exit 0
            fi                
            F_Name=${FORWARD_LINE:1:${#FORWARD_LINE}-3}
            R_Name=${REVERSE_LINE:1:${#REVERSE_LINE}-3}                
            if [[ $F_Name != $R_Name ]]; then
                echo -e "Record names do not match. "
                echo -e "Line ${line_iterator}: ${FORWARD_LINE}"
                echo -e "Line ${line_iterator}: ${REVERSE_LINE}"
                exit 0
            fi                
            line_iterator=$(( $line_iterator + 1 ))            
        else 
            if [[ ! ( $FORWARD_LINE =~ ^[ATCGNatcgn]+$ ) ]]; then
                echo -e "Ambigous sequence detected for read ${read_iterator} at line ${line_iterator} in file ${F_READS}"
                exit 0
            fi                
            read_iterator=$(( $read_iterator + 1 ))
            line_iterator=$(( $line_iterator + 1 ))            
        fi
        unset FORWARD_LINE
        unset REVERSE_LINE
    done        
    echo -e "$line_iterator lines and $read_iterator reads"
    echo -e "No errors detected."
    echo -e ""
}
export -f fastaConsistencyChecker
FILE3="filepath/file1"    
FILE4="filepath/file2"    
fastaConsistencyChecker $FILE3 $FILE4 
</code></pre>