<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Week 3 lecture notes
Exercise 2 review - common mistakes
Including directories in paths
If you create a file in a lower directory, then want to modify, move, or delete it, you have to use the directory to refer to it.
Step1:  ">" vs ">>"
Both ">" and ">>" redirect output from the screen to a file.  Both will create new files if none yet exists.  Only ">" will overwrite an existing file; ">>" will append to an existing file.
Step2:  lower|sort|uniq or sort|lower|uniq
Order matters!  Consider the text from exercise-02.
Step3:  Among the set of three functions
Step4:  More about grep
grep is a lot more powerful than what you've seen so far.  More than anything else, it's commonly used to find text within files.  For example, to find lines with "Romeo" in Romeo and Juliet
Step5:  There are many, many options, such as case-insensitivity
Step6:  Another useful one is to print line numbers for matching lines
Step7:  We can also negate certain terms - show non-matches.
Step8:  And one more useful tip is to match more than one thing
Step9:  Wildcards with "*"
Sometimes you need to perform a task with a set of files that share a characteristic like a file extension.  The shell lessons had examples with .pdb files.  This is common.
The * (asterisk, or just "star") is a wildcard, which matches zero-to-many characters.
Step10:  The ? (question mark) is a wildcard that matches exactly one character.
Step11:  The difference is subtle - these two would have worked interchangeably on the above.  But note
Step12:  See the difference?  The * can match more than one character; ? only matches one. 
Writing Python filters
Starting with the samplefilter.py filter, let's write some of our own.
Step13:  Working with GNU Parallel
GNU Parallel is an easy to use but very powerful tool with a lot of options.  You can use it to process a lot of data easily and it can also make a big mess in a hurry.  For more examples, see the tutorial page.
Let's start with something we've seen before
Step14:  That's 25,875 lines and 218,062 words in the texts of Romeo and Juliet and Little Women.
We can split them up into word counts one at a time like we did in exercise-02
Step15:  Note that I've wrapped lines around by using the \ character.  To me, this looks easier to read - you can see each step of the pipeline one at a time.  The \ only means "this shell line continues on the next line".  The | still acts as the pipe.
Let's look at a second book, Little Women.  We'll add time to get a sense of how long it takes.
Step16:  It looks like Little Women is much longer, which makes sense - it's a novel, not a play.  More text!
To compare the two directly
Step17:  We can run through both files at once by giving both file names to grep
Step18:  Do those numbers look right?  
Let's take a closer look at what's going on.
Step19:  Aha!  grep is not-so-helpfully including the second filename on the lines matched from the second file, but not on the first.  That's why the counts are off.
There's probably an option to tell grep not to do that.  But let's try something completely different.
First, let's break the step into the data parallel piece.  For which part of this pipeline is completely data parallel?
Step20:  See what we did there?  We parallelized the data, then brought it back together for the rest of the pipeline.
Let's try it on a much bigger dataset.  (Note that we're unzipping into a new directory with unzip -d.)
<ASSISTANT_TASK:>
Python Code:
!mkdir mydirectory
!ls > mydirectory/myfiles.txt
!rm myfiles.txt
!rm mydirectory/myfiles.txt
!ls mydirectory
Explanation: Week 3 lecture notes
Exercise 2 review - common mistakes
Including directories in paths
If you create a file in a lower directory, then want to modify, move, or delete it, you have to use the directory to refer to it.
End of explanation
!date > datefile.txt
!cat datefile.txt
!date > datefile.txt
!cat datefile.txt
!date >> datefile.txt
!date >> datefile.txt
!cat datefile.txt
Explanation: ">" vs ">>"
Both ">" and ">>" redirect output from the screen to a file.  Both will create new files if none yet exists.  Only ">" will overwrite an existing file; ">>" will append to an existing file.
End of explanation
!wget https://github.com/gwsb-istm-6212-fall-2016/syllabus-and-schedule/raw/master/exercises/pg2500.txt
!grep -oE '\w{{2,}}' pg2500.txt | grep -v '^[0-9]' | uniq -c | head
Explanation: lower|sort|uniq or sort|lower|uniq
Order matters!  Consider the text from exercise-02.
End of explanation
!grep -oE '\w{{2,}}' pg2500.txt | grep -v '^[0-9]' | uniq -c | tr '[:upper:]' '[:lower:]' | sort | head
!grep -oE '\w{{2,}}' pg2500.txt | grep -v '^[0-9]' | uniq -c | sort | tr '[:upper:]' '[:lower:]' | head
!grep -oE '\w{{2,}}' pg2500.txt | grep -v '^[0-9]' | sort | tr '[:upper:]' '[:lower:]' | uniq -c | head
!grep -oE '\w{{2,}}' pg2500.txt | grep -v '^[0-9]' | sort | uniq -c | tr '[:upper:]' '[:lower:]' | head
!grep -oE '\w{{2,}}' pg2500.txt | grep -v '^[0-9]' | tr '[:upper:]' '[:lower:]' | sort | uniq -c | head
!grep -oE '\w{{2,}}' pg2500.txt | grep -v '^[0-9]' | tr '[:upper:]' '[:lower:]' | uniq -c | sort | head 
Explanation: Among the set of three functions: {uniq, lower, sort} there are six orderings.  Which produce correct results, and why?
uniq, lower, sort
uniq, sort, lower
sort, lower, uniq
sort, uniq, lower
lower, sort, uniq
lower, uniq, sort
End of explanation
!grep Romeo romeo.txt | head
Explanation: More about grep
grep is a lot more powerful than what you've seen so far.  More than anything else, it's commonly used to find text within files.  For example, to find lines with "Romeo" in Romeo and Juliet:
End of explanation
!grep -i what romeo.txt | head
Explanation: There are many, many options, such as case-insensitivity:
End of explanation
!grep -n Juliet romeo.txt | head
Explanation: Another useful one is to print line numbers for matching lines:
End of explanation
!grep -n Juliet romeo.txt | grep -v Romeo | head
Explanation: We can also negate certain terms - show non-matches.
End of explanation
!grep "Romeo\|Juliet" romeo.txt | head
Explanation: And one more useful tip is to match more than one thing:
End of explanation
!ls *.txt
Explanation: Wildcards with "*"
Sometimes you need to perform a task with a set of files that share a characteristic like a file extension.  The shell lessons had examples with .pdb files.  This is common.
The * (asterisk, or just "star") is a wildcard, which matches zero-to-many characters.
End of explanation
!cp romeo.txt womeo.txt
!ls ?omeo.txt
!ls wome?.txt
Explanation: The ? (question mark) is a wildcard that matches exactly one character.
End of explanation
!ls wo*.txt
!ls wo?.txt
Explanation: The difference is subtle - these two would have worked interchangeably on the above.  But note:
End of explanation
!chmod +x simplefilter.py
!head pg2500.txt | ./simplefilter.py
!cp simplefilter.py lower.py
!head pg2500.txt | ./lower.py
Explanation: See the difference?  The * can match more than one character; ? only matches one. 
Writing Python filters
Starting with the samplefilter.py filter, let's write some of our own.
End of explanation
!wc *.txt
Explanation: Working with GNU Parallel
GNU Parallel is an easy to use but very powerful tool with a lot of options.  You can use it to process a lot of data easily and it can also make a big mess in a hurry.  For more examples, see the tutorial page.
Let's start with something we've seen before:  splitting a text file up and counting its unique words.
End of explanation
!grep -oE '\w{{2,}}' romeo.txt \
    | tr '[:upper:]' '[:lower:]' \
    | sort \
    | uniq -c \
    | sort -rn \
    | head -10
Explanation: That's 25,875 lines and 218,062 words in the texts of Romeo and Juliet and Little Women.
We can split them up into word counts one at a time like we did in exercise-02:
End of explanation
!time grep -oE '\w{{2,}}' women.txt \
    | tr '[:upper:]' '[:lower:]' \
    | sort \
    | uniq -c \
    | sort -rn \
    | head -10
Explanation: Note that I've wrapped lines around by using the \ character.  To me, this looks easier to read - you can see each step of the pipeline one at a time.  The \ only means "this shell line continues on the next line".  The | still acts as the pipe.
Let's look at a second book, Little Women.  We'll add time to get a sense of how long it takes.
End of explanation
!wc *.txt
Explanation: It looks like Little Women is much longer, which makes sense - it's a novel, not a play.  More text!
To compare the two directly:
End of explanation
!time grep -oE '\w{{2,}}' romeo.txt women.txt \
    | tr '[:upper:]' '[:lower:]' \
    | sort \
    | uniq -c \
    | sort -rn \
    | head -10
Explanation: We can run through both files at once by giving both file names to grep:
End of explanation
!time grep -oE '\w{{2,}}' romeo.txt women.txt \
    | tr '[:upper:]' '[:lower:]' \
    | sort \
    | uniq -c \
    | grep "and" \
    | tail -10
Explanation: Do those numbers look right?  
Let's take a closer look at what's going on.
End of explanation
!time ls *.txt \
    | parallel -j+0 "grep -oE '\w{2,}' {} | tr '[:upper:]' '[:lower:]' >> all-words.txt"
!time sort all-words.txt \
    | uniq -c \
    | sort -rn \
    | head -10
Explanation: Aha!  grep is not-so-helpfully including the second filename on the lines matched from the second file, but not on the first.  That's why the counts are off.
There's probably an option to tell grep not to do that.  But let's try something completely different.
First, let's break the step into the data parallel piece.  For which part of this pipeline is completely data parallel?
End of explanation
!unzip -d many-texts texts.zip
!ls -l many-texts | wc -l
!wc many-texts/*.txt
!time ls many-texts/*.txt \
    | parallel --eta -j+0 "grep -oE '\w{2,}' {} | tr '[:upper:]' '[:lower:]' >> many-texts/all-words.txt"
!time sort many-texts/all-words.txt \
    | uniq -c \
    | sort -rn \
    | head -10
Explanation: See what we did there?  We parallelized the data, then brought it back together for the rest of the pipeline.
Let's try it on a much bigger dataset.  (Note that we're unzipping into a new directory with unzip -d.)
End of explanation
<END_TASK>