<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Extracting text from PDFs
PDF stands for "Portable Document Format" and PDF files contain images, text, and page layout information.  PDF files are actually programs in a very simple programming language and, hence, can display just about anything. Much of what you see inside a PDF file is text, however, and we can grab that text without the layout information using poppler. (I used to use pdfminer but somehow no longer works on OS X.) Install it with
Step1:  That command downloads the file and because of the redirection operator, &gt;, the output gets written to tsla.pdf up in /tmp directory.
Once we have the data, we can pass the filename to pdftotext to extract the text
Step2:  (Don't worry about those warnings.)
Step3:  Once you have text output, you can perform whatever analysis you'd like without having to worry about the data coming in PDF form. For example, you might want to run some analysis on financial documents but they are all in PDF. First, convert to text and then perform your analysis.
Exercise
Read that text file with a Python script and split the document into a list of words. Print out the first 100 words. It should look like
Step4:  Text processing from the command line
It's often the case that we can do a huge amount of cleanup on unstructured text before using Python to process it more formally. We can delete unwanted characters, squeeze repeated characters, reformat, etc... In this section you will do a number of exercises that get you use to processing files from the command line. If you'd like to dig further, you can see this link.
The operating system launches all commands in a pipeline sequence as separate processes, which means they can run on multiple processors simultaneously. This gives us parallel processing without having to write complicated code. As data is completed by one stage, it passes it to the next stage of the pipeline, and continues to work on its input. The next stage consumes that input in parallel. Consequently, processing text from the command line can be extremely efficient, much more so than doing it in Python.
Exercise
Using the tr (translate) command from the terminal, strip all of the new lines from the text file you created above (/tmp/tsla.txt).  Look at the manual page with this command
Step5:  Exercise
Reformat the text using tr and fold. The fold command wraps lines at 80 characters; use its -s option to making break lines at spaces between words.
Step6:  Exercise
It is sometimes useful to put a line number at the left edge of all lines. For example, you might want to create a unique ID number for each row of a CSV file. Pipe the output of the previous command to nl so that you get the line number on the left edge.
Step7:  Exercise
Convert the text to all lowercase using tr. Hint
Step8:  Exercise
Do the same thing but on the text that has the new lines removed.
<ASSISTANT_TASK:>
Python Code:
! curl https://www.tesla.com/sites/default/files/tesla-model-s.pdf > /tmp/tsla.pdf
Explanation: Extracting text from PDFs
PDF stands for "Portable Document Format" and PDF files contain images, text, and page layout information.  PDF files are actually programs in a very simple programming language and, hence, can display just about anything. Much of what you see inside a PDF file is text, however, and we can grab that text without the layout information using poppler. (I used to use pdfminer but somehow no longer works on OS X.) Install it with:
bash
brew install poppler
or
bash
brew upgrade poppler
Then use pdftotext as a command from the commandline, which will extract out the text  and save in a text file. First download a sample PDF, such as Tesla model S, which we can easily do from the command line using curl (which you might have to install):
End of explanation
! pdftotext /tmp/tsla.pdf # saves into /tmp/tsla.txt
Explanation: That command downloads the file and because of the redirection operator, &gt;, the output gets written to tsla.pdf up in /tmp directory.
Once we have the data, we can pass the filename to pdftotext to extract the text:
End of explanation
! head -10 /tmp/tsla.txt
Explanation: (Don't worry about those warnings.)
End of explanation
with open('/tmp/tsla.txt') as f:
    print(f.read().split()[:100])
Explanation: Once you have text output, you can perform whatever analysis you'd like without having to worry about the data coming in PDF form. For example, you might want to run some analysis on financial documents but they are all in PDF. First, convert to text and then perform your analysis.
Exercise
Read that text file with a Python script and split the document into a list of words. Print out the first 100 words. It should look like:
['Model', 'S', 'Premium', 'Electric', 'Sedan', 'An', ...]
End of explanation
! tr -s '\n' ' ' < /tmp/tsla.txt | head -c 200
Explanation: Text processing from the command line
It's often the case that we can do a huge amount of cleanup on unstructured text before using Python to process it more formally. We can delete unwanted characters, squeeze repeated characters, reformat, etc... In this section you will do a number of exercises that get you use to processing files from the command line. If you'd like to dig further, you can see this link.
The operating system launches all commands in a pipeline sequence as separate processes, which means they can run on multiple processors simultaneously. This gives us parallel processing without having to write complicated code. As data is completed by one stage, it passes it to the next stage of the pipeline, and continues to work on its input. The next stage consumes that input in parallel. Consequently, processing text from the command line can be extremely efficient, much more so than doing it in Python.
Exercise
Using the tr (translate) command from the terminal, strip all of the new lines from the text file you created above (/tmp/tsla.txt).  Look at the manual page with this command:
python
$ man tr
You can pipe the output of tr to head -c 150 to only print out the first 150 characters of the output.
End of explanation
! tr -s '\n' ' ' < /tmp/tsla.txt | fold -s | head -10
Explanation: Exercise
Reformat the text using tr and fold. The fold command wraps lines at 80 characters; use its -s option to making break lines at spaces between words.
End of explanation
! tr -s '\n' ' ' < /tmp/tsla.txt | fold -s | nl | head -10
Explanation: Exercise
It is sometimes useful to put a line number at the left edge of all lines. For example, you might want to create a unique ID number for each row of a CSV file. Pipe the output of the previous command to nl so that you get the line number on the left edge.
End of explanation
! tr 'A-Z' 'a-z' < /tmp/tsla.txt | head -c 150
Explanation: Exercise
Convert the text to all lowercase using tr. Hint: a-z and A-Z are regular expressions that describe English characters and uppercase English characters.
End of explanation
! tr -s '\n' ' ' < /tmp/tsla.txt | tr 'A-Z' 'a-z' | head -c 150
Explanation: Exercise
Do the same thing but on the text that has the new lines removed.
End of explanation
<END_TASK>