<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Processing a LexisNexus text export into CSV
Preparation
download the file
Step1:  Show the number of characters in the text file
Step2:  Downloading the text file directly from github using the <code>requests</code> module
Step3:  The <code>get</code> function takes a URL and returns the content at that address.
Step4:  Examine the first 20,000 characters to find string patterns that mark divisions between documents
Step5:  The string <code>of 1000 DOCUMENTS</code> looks like a good candidate for splitting the the text file into the individual documents
Split the text string into n chunks using of 1000 DOCUMENTS
Step6:  Python excurcus
Step7:  Another, 'less Pythonic', way to do this is to create a loop that uses the indices of each item in the list, e.g.
Step8:  But often you want to have both each item in the list and its index without having to do list[idx] to get the item. The enumerate function helps in such cases.
enumerate(list) returns a list of tuples, where each item in the list consists of a pair where the first item is the index and second the item itself.
Step9:  Back to the LexisNexus task
We can use the enumerate and for loop idiom to check how good our proposed strings are for splitting the document up into components.
For example, it looks like the string ( END ) could be a good marker of the end of the body. So we can test which documents contain it using the count function and testing whether we get at least one instance
Step10:  AH!  - actually doesn't look like a great candidate for splitting a document in to body and post body sections.
A second look at some sample documents suggests we might be able to use LOAD-DATE instead.
Step11:  We can use list interpolation to get a count of the number of documents that contain a feature
Step12:  Now we have a set of string markers and a strategy for splitting documents up
Step13:  Now we have that figured out we can set two variables
Step14:  Then we can get the three parts of the document we want
Step15:  Python excurcus
Step16:  Back to the LexisNexus task
We are going to use a list of dictionaries approach to store the prebody, body, postbody components for each document
Step17:  Python excursus
Step18:  The final parts of the task!
Now we want to write out the documents split into three parts to a CSV file
Then just for fun we construction a frequency list of all the words in the documents
Step19:  Frequency counts
The <code>Counter</code> function generates a dictionary-like object with words as keys and the number of times they occur in a sequence as values
It is a quick way to generate a word frequency list from a set of tokenized documents (i.e., where the text has been turned into a list of words)
Step20:  Simple example of using <code>Counter</code>
<code>Counter</code> works by passing it a list of items, e.g.
Step21:  and it returns a dictionary with a count for the number of times each item occurs
Step22:  Just like a dictionary you can get the frequency for a specific item like this
Step23:  <code>Counter</code> object has an <code>update</code> method that allows multiple lists to be counted.
Step24:  First define a new <code>Counter</code>
Step25:  Then update it with the words from text1
Step26:  Then update it again with the words from text2
Step27:  A simple example of looping of some texts and generating a frequency list using Counter
Step28:  Finally lets make the formatting a bit more pretty by looping over the frequency list and producing a tab separated table
Step29:  Counting words in the LexisNexus documents
For a single LexisNexus document loaded in from the CSV file (the list of dictionaries), we select the document by index and then the body component
Step30:  Find the frequency of the word vaping
Step31:  Finally we can create a frequency list for all the documents with a loop and using the update function on the Counter object.
<ASSISTANT_TASK:>
Python Code:
text = open('data/LexisNexusVapingExample.txt', 'r').read()
Explanation: Processing a LexisNexus text export into CSV
Preparation
download the file: https://github.com/mbod/intro_python_for_comm/blob/master/data/LexisNexusVapingExample.txt
place it in the data folder of your IPython notebook
Task
Load the text file <code>LexisNexusVapingExample.txt</code> into a variable text
Examine the first 20000 chars to figure out how articles are separated
Create a list by splitting on the separator string
Seperate each article into prebody, body and postbody components
Save the output to a CSV file with three columns:
   prebody, body and postbody
   and a row for each article
Loading contexts of the text file from the data folder
If you downloaded the text file and placed in the data folder you can read it into a variable like this:
End of explanation
len(text)    
Explanation: Show the number of characters in the text file:
End of explanation
import requests     # import the module
Explanation: Downloading the text file directly from github using the <code>requests</code> module
End of explanation
resp = requests.get('https://raw.githubusercontent.com/mbod/intro_python_for_comm/master/data/LexisNexusVapingExample.txt')
text2=resp.text    # assign content of response to a variable
text2[:300]   # show first 300 characters
print(text2[:300])   # use print to see formatting (spacing and newlines etc.)
Explanation: The <code>get</code> function takes a URL and returns the content at that address.
End of explanation
print(text[0:20000])
Explanation: Examine the first 20,000 characters to find string patterns that mark divisions between documents
End of explanation
chunks = text.split('of 1000 DOCUMENTS')
len(chunks)    # see how many chunks this produces
docs = chunks[1:]
Explanation: The string <code>of 1000 DOCUMENTS</code> looks like a good candidate for splitting the the text file into the individual documents
Split the text string into n chunks using of 1000 DOCUMENTS:
End of explanation
alist = [1,2,3,4,5]
slist = ['a','b','c','d']
for item in alist:
    print(item)
    
for item in slist:
    print('The current item is:',item)
Explanation: Python excurcus:  Using enumerate to loop over lists
When you have a list of items and what to process each in turn then using a for loop is a common approach, e.g.
End of explanation
for idx in range(0,len(alist)):
    print('Index', idx, 'is item:', alist[idx])
Explanation: Another, 'less Pythonic', way to do this is to create a loop that uses the indices of each item in the list, e.g.
End of explanation
list(enumerate(slist))
result = list(enumerate(slist))
result[0]
for idx, item in enumerate(slist):
    print(idx, item)
Explanation: But often you want to have both each item in the list and its index without having to do list[idx] to get the item. The enumerate function helps in such cases.
enumerate(list) returns a list of tuples, where each item in the list consists of a pair where the first item is the index and second the item itself.
End of explanation
for idx, doc in enumerate(docs):
    print('Document', idx, 'has ( END )?', doc.count('( END )') > 0)
Explanation: Back to the LexisNexus task
We can use the enumerate and for loop idiom to check how good our proposed strings are for splitting the document up into components.
For example, it looks like the string ( END ) could be a good marker of the end of the body. So we can test which documents contain it using the count function and testing whether we get at least one instance:
End of explanation
for idx, doc in enumerate(docs):
    print('Document', idx, 'has LOAD-DATE?', doc.count('LOAD-DATE:') > 0)
Explanation: AH!  - actually doesn't look like a great candidate for splitting a document in to body and post body sections.
A second look at some sample documents suggests we might be able to use LOAD-DATE instead.
End of explanation
has_end = sum([doc.count('( END )')>0 for doc in docs])
print(has_end, 'documents with ( END ) out of ', len(docs), 'docs')
has_load_date = sum([doc.count('LOAD-DATE:')>0 for doc in docs])
print(has_load_date, 'documents with LOAD-DATE out of ', len(docs), 'docs')
Explanation: We can use list interpolation to get a count of the number of documents that contain a feature
End of explanation
doc = docs[0]
doc.index('LENGTH')   # find the character position for the start of string LENGTH
doc[224:274]    # slice the string starting at this point plus 50 characters
doc.index('\n',224)  # find the first newline character after the start of LENGTH
doc[241:280]   # slice after this character
Explanation: Now we have a set of string markers and a strategy for splitting documents up:
for each document
split into three parts
prebody = text up to LENGTH xxx words
body = text from LENGTH to before LOAD-DATE
postbody = LOAD-DATE to the end
End of explanation
start_pos = doc.index('LENGTH')
start_pos = doc.index('\n', start_pos)
end_pos = doc.index('LOAD-DATE:')
Explanation: Now we have that figured out we can set two variables:
start_pos for the beginning of the body (the line after the one beginning with LENGTH)
end_pos the point were LOAD-DATE begins
End of explanation
pre_body=doc[:start_pos]
body = doc[start_pos:end_pos]
post_body = doc[end_pos:]
Explanation: Then we can get the three parts of the document we want
End of explanation
exdict = { 'item1': 123, 'item2': 'asdsad', 'item3': [1,2,3,'asd','dada'] }   # define a dictionary
exdict
exdict['item2']   # get the value associated with the key 'item2'
# add a new value associated with the key 'item4' which is itself a dictionary
exdict['item4'] = {'a': 12323, 'b': [1,2,23]} 
exdict
exdict['item4']['a']   # address this dictionary of dictionary structure
exdict['item3'][3]
Explanation: Python excurcus:  Dictionaries
Alongside lists one of the most useful structures in Python is a dictionary. It is an unordered set of key-value pairings.
Data is organized and identified by a unique key using the syntax 'key' : value
End of explanation
doc_dict = {'prebody': pre_body, 'body': body, 'postbody': post_body}
doc_dict['postbody']
rows = []
for idx, doc in enumerate(docs):
    
    try:
        start_pos = doc.index('LENGTH')
        start_pos = doc.index('\n', start_pos)
        end_pos = doc.index('LOAD-DATE:')
    except:
        print('ERROR with doc', idx)
        continue
    
    doc_dict = {
        'prebody': doc[:start_pos],
        'body': doc[start_pos: end_pos],
        'postbody': doc[end_pos:]
    }
    
    rows.append(doc_dict)
print(docs[13])
Explanation: Back to the LexisNexus task
We are going to use a list of dictionaries approach to store the prebody, body, postbody components for each document
End of explanation
import this
Explanation: Python excursus: The Zen of Python
A little bit of poetry from the creators of Python explaining the design and suggestions for truly Pythonic coding!
End of explanation
import csv
with open('data/articles.csv','w') as out:
    csvfile = csv.DictWriter(out, fieldnames=('prebody','body','postbody'))
    csvfile.writeheader()
    csvfile.writerows(rows)
docs2 = [r for r in csv.DictReader(open('data/articles.csv','r'))]
len(docs2)
print(docs2[0])
Explanation: The final parts of the task!
Now we want to write out the documents split into three parts to a CSV file
Then just for fun we construction a frequency list of all the words in the documents
End of explanation
from collections import Counter
Explanation: Frequency counts
The <code>Counter</code> function generates a dictionary-like object with words as keys and the number of times they occur in a sequence as values
It is a quick way to generate a word frequency list from a set of tokenized documents (i.e., where the text has been turned into a list of words)
End of explanation
count = Counter(['a','a','v','c','d','e','a','c'])
Explanation: Simple example of using <code>Counter</code>
<code>Counter</code> works by passing it a list of items, e.g.:
End of explanation
count.items()
Explanation: and it returns a dictionary with a count for the number of times each item occurs:
End of explanation
count['a']   # how many times does 'a' occur in the list ['a','a','v','c','d','e','a','c']
Explanation: Just like a dictionary you can get the frequency for a specific item like this:
End of explanation
text1 = 'This is a text with some words in it'
text2 = 'This is another text with more words that the other one has in it'
tokens1 = text1.lower().split()
tokens2 = text2.lower().split()
print('text1:', tokens1)
print('text2:', tokens2)
Explanation: <code>Counter</code> object has an <code>update</code> method that allows multiple lists to be counted.
End of explanation
freq = Counter()
Explanation: First define a new <code>Counter</code>
End of explanation
freq.update(tokens1)
freq.items()
Explanation: Then update it with the words from text1
End of explanation
freq.update(tokens2)
freq.items()
Explanation: Then update it again with the words from text2
End of explanation
texts = [
    'This is the first text and it has words',
    'This is the second text and it has some more words',
    'Finally this one has the most words of all three examples words and words and words'
]
freq2 = Counter()
for text in texts:
    # turn the text into lower case and split on whitespace
    tokens = text.lower().split()
    freq2.update(tokens)
    
# show the top 10 most frequent words
print(freq2.most_common(10))
Explanation: A simple example of looping of some texts and generating a frequency list using Counter
End of explanation
for item in freq2.most_common(7):
    print("{}\t\t{}".format(item[0],item[1]))
Explanation: Finally lets make the formatting a bit more pretty by looping over the frequency list and producing a tab separated table:
End of explanation
freq_list = Counter(docs2[0]['body'].lower().split())
freq_list.most_common()
Explanation: Counting words in the LexisNexus documents
For a single LexisNexus document loaded in from the CSV file (the list of dictionaries), we select the document by index and then the body component:
End of explanation
freq_list['vaping']
Explanation: Find the frequency of the word vaping
End of explanation
freq_list_all = Counter()
for doc in docs2:
    body_text = doc['body']
    tokens = body_text.lower().split()
    freq_list_all.update(tokens)
    
print(freq_list_all.most_common())
Explanation: Finally we can create a frequency list for all the documents with a loop and using the update function on the Counter object.
End of explanation
<END_TASK>