<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 SecLists Index parsing
Example
Step1:  The generated code from seclists.org contains an unterminated anchor tag, so to make things easier for BeautifulSoup's parser, we'll just replace this manually. This particular tag is a good tag to use as a locator for the messages section, so it's good to make sure that it's valid html.
Our soup "query" basically consists of
Step2:  We end up with an array of &lt;li&gt; tags, but note that children tags are encoded as embedded &lt;ul&gt; portions. 
Example
Step3:  The index file summarizes the reply's details, but we would like the full author with email, and a complete timestamp. To do this, we'll need to delve into the actual raw.html. For this notebook, we'll download the file again, but in actual usage, we would just open the existing file. 
The full author and timestamp are contained between the X-Head-of-Message comments.
Step4:  Finally, let's put this data into csv format.
<ASSISTANT_TASK:>
Python Code:
import requests
from bs4 import BeautifulSoup
from IPython.display import Pretty
import pprint
pp = pprint.PrettyPrinter(indent=4)
url = 'http://seclists.org/fulldisclosure/2017/Jan'
r = requests.get(url)
raw = r.text
Pretty(raw)
Explanation: SecLists Index parsing
Example: http://seclists.org/fulldisclosure/2017/Jan
We will do basic parsing of the index file, mainly to extract the thread hierarchy contained within the &lt;ul&gt; tags. 
Normally, we have the file already downloaded via seclists_crawler_raw.py, but for this notebook, we'll download a fresh copy.
End of explanation
raw = raw.replace('<a name="begin">', '<a name="begin"></a>')
soup = BeautifulSoup(raw, 'html5lib')
begin = soup.find(attrs={'name':'begin'}) #beginning of msg links
items = begin.find_next('ul').find_all('li', recursive=False)
pp.pprint(items)
Explanation: The generated code from seclists.org contains an unterminated anchor tag, so to make things easier for BeautifulSoup's parser, we'll just replace this manually. This particular tag is a good tag to use as a locator for the messages section, so it's good to make sure that it's valid html.
Our soup "query" basically consists of:
1. Find me the tag with name = "begin"
2. Find me the first ul after this tag
End of explanation
import re
def read_messages(items, messages, idroot, parent):
    for li in items:
        msg = li.find('a')
        if msg == None:
            #some messages just read "Possible follow-ups" with no link--skip
            continue
        id = idroot + msg['href']
        title = msg.text
        
        whowhen = li.find('em').text
        rx = re.compile('(.+) \((.+)\)')
        m = rx.search(whowhen)
        who = m.group(1)
        when = m.group(2)
        messages.append({
            'index': msg['href'],
            'id': id,
            'title': title,
            'parent': parent,
            'author': who,
            'date': when
        })
        
        replies = li.find('ul')
        if replies != None:
            read_messages(replies.find_all('li', recursive=False), messages, idroot, id)
    return messages
messages = []
idroot = '2017_Jan_'
read_messages(items, messages, idroot, None)
pp.pprint(messages)
Explanation: We end up with an array of &lt;li&gt; tags, but note that children tags are encoded as embedded &lt;ul&gt; portions. 
Example:
&lt;li&gt;&lt;a href="3" name="3"&gt;Persisted Cross-Site Scripting (XSS) in Confluence Jira   Software&lt;/a&gt; &lt;em&gt;jlss (Jan 03)&lt;/em&gt;
     &lt;ul&gt;
         &lt;li&gt;&lt;a href="9" name="9"&gt;Re: Persisted Cross-Site Scripting (XSS) in Confluence Jira Software&lt;/a&gt; &lt;em&gt;Moritz Naumann (Jan 04)&lt;/em&gt;
             &lt;ul&gt;
                 &lt;li&gt;&lt;a href="12" name="12"&gt;Re: Persisted Cross-Site Scripting (XSS) in Confluence Jira Software&lt;/a&gt; &lt;em&gt;jlss (Jan 06)&lt;/em&gt;
                 &lt;/li&gt;
We'll have to use recursion to be able to run through each layer, so lets define a function
Args:
* items: &lt;li&gt; tags at our given thread level
* messages: array used to hold our results
* idroot: 
* parent: id of the parent message
End of explanation
import pendulum
message = messages[4]
reply_url = url + '/' + message['index']
r = requests.get(reply_url)
reply = r.text
start = reply.index('<!--X-Head-of-Message-->') + 24
end = reply.index('<!--X-Head-of-Message-End-->')
head = reply[start:end]
soup = BeautifulSoup(head, 'html5lib')
ems = soup.find_all('em')
for em in ems:
    if em.text == 'From':
        author = em.next_sibling
        #list obfuscates email by replacing @ with ' () ' and removing periods from domain name
        if author.startswith(': '):
            author = author[2:]
        author = author.replace(' () ', '@')
        at = author.find('@')
        author = author[:at] + author[at:].replace(' ', '.') 
        message['author'] = author
    elif em.text == 'Date':
        date = em.next_sibling
        if date.startswith(': '):
            date = date[2:]
        message['date'] = str(pendulum.parse(date).in_timezone('UTC'))
print(message)
Explanation: The index file summarizes the reply's details, but we would like the full author with email, and a complete timestamp. To do this, we'll need to delve into the actual raw.html. For this notebook, we'll download the file again, but in actual usage, we would just open the existing file. 
The full author and timestamp are contained between the X-Head-of-Message comments.
End of explanation
import csv
import sys
output = csv.writer(sys.stdout)
output.writerow(['id', 'title', 'date', 'author', 'parent'])
for x in messages:
    output.writerow([x['id'],
                     x['title'],
                     x['date'],
                     x['author'],
                     x['parent']])
Explanation: Finally, let's put this data into csv format.
End of explanation
<END_TASK>