<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 A simple (ie. no error checking or sensible engineering) notebook to extract the student answer data from a single xml file. 
I'll also export the data to a csv file at the end of this, so that it's easy to read in at the beginning of another notebook.
Following discussions with Suraj, we want the representation to take into account the student's response, the official answer, and the grade. So there'll be a little fiddliness linking the student response back to the gold standard response.
So, first read the file
Step1:  It's an xml file, so we'll need the xml.etree parser, and pandas so that we can import into a dataframe
Step2:  Now, the reference answers are in the second daughter node of the tree. We can extract these and store them in a dictionary. To distinguish between reference answer tokens and student response tokens, I'm going to append each token in the reference answers with <code>_RA</code>, and each of the tokens in a student response with <code>_SR</code>.
Step3:  Next, we need to extract each of the student responses. These are in the third daughter node
Step4:  OK, that seems to work OK. Now, let's define a function that takes a filename, and returns the list of token dictionaries
Step5:  We now have a function which takes a filename and returns a list of tokenised student responses and reference answers
Step6:  So next we need to be able to build a document frequency dictionary from a list of tokenised documents.
Step7:  Next, define a function which takes a list of tokens and a document frequency dictionary, and returns a dictionary of the tf.idf values for each of the tokens in the list. Note
Step8:  Finally, we want to convert the outputs for all of the responses into a dataframe.
Step9:  Cool, that seems to work. Now just need to do it for the complete set of files. Just use beetle/train/core for the time being.
Step10:  Use os.walk to get the files
Step11:  We can now do the same as before, but this time using all the files to construct the final dataframe. We also need a series containing the accuracy measures.
Step12:  And finish by exporting to a csv file
<ASSISTANT_TASK:>
Python Code:
filename='semeval2013-task7/semeval2013-Task7-5way/beetle/train/Core/FaultFinding-BULB_C_VOLTAGE_EXPLAIN_WHY1.xml'
Explanation: A simple (ie. no error checking or sensible engineering) notebook to extract the student answer data from a single xml file. 
I'll also export the data to a csv file at the end of this, so that it's easy to read in at the beginning of another notebook.
Following discussions with Suraj, we want the representation to take into account the student's response, the official answer, and the grade. So there'll be a little fiddliness linking the student response back to the gold standard response.
So, first read the file:
End of explanation
import pandas as pd
from xml.etree import ElementTree as ET
tree=ET.parse(filename)
r=tree.getroot()
Explanation: It's an xml file, so we'll need the xml.etree parser, and pandas so that we can import into a dataframe:
End of explanation
from string import punctuation
def to_tokens(textIn):
    '''Convert the input textIn to a list of tokens'''
    tokens_ls=[t.lower().strip(punctuation) for t in textIn.split()]
    # remove any empty tokens
    return [t for t in tokens_ls if t]
str='"Help!" yelped the banana, who was obviously scared out of his skin.'
print(str)
print(to_tokens(str))
refAnswers_dict={refAnswer.attrib['id']:[t+'_RA' for t in to_tokens(refAnswer.text)] 
                 for refAnswer in r[1]}    
refAnswers_dict
Explanation: Now, the reference answers are in the second daughter node of the tree. We can extract these and store them in a dictionary. To distinguish between reference answer tokens and student response tokens, I'm going to append each token in the reference answers with <code>_RA</code>, and each of the tokens in a student response with <code>_SR</code>.
End of explanation
print(r[2][0].text)
r[2][0].attrib
responses_ls=[]
for (i, studentResponse) in enumerate(r[2]):
    if 'answerMatch' in studentResponse.attrib:
        matchTokens_ls=refAnswers_dict[studentResponse.attrib['answerMatch']]
    else:
        matchTokens_ls=[]
    responses_ls.append({'accuracy':studentResponse.attrib['accuracy'],
                         'text':studentResponse.text,
                         'tokens':[t+'_SR' for t in to_tokens(studentResponse.text)] + matchTokens_ls})
responses_ls[36]
Explanation: Next, we need to extract each of the student responses. These are in the third daughter node:
End of explanation
def extract_token_dictionaries(filenameIn):
    
    # Localise the to_tokens function
    def to_tokens_local(textIn):
        '''Convert the input textIn to a list of tokens'''
        tokens_ls=[t.lower().strip(punctuation) for t in textIn.split()]
        # remove any empty tokens
        return [t for t in tokens_ls if t]
    tree=ET.parse(filenameIn)
    root=tree.getroot()
    
    refAnswers_dict={refAnswer.attrib['id']:[t+'_RA' for t in to_tokens_local(refAnswer.text)]
                     for refAnswer in root[1]}
    responsesOut_ls=[]
    for (i, studentResponse) in enumerate(root[2]):
        if 'answerMatch' in studentResponse.attrib:
            matchTokens_ls=refAnswers_dict[studentResponse.attrib['answerMatch']]
        else:
            matchTokens_ls=[]
        responsesOut_ls.append({'accuracy':studentResponse.attrib['accuracy'],
                                'text':studentResponse.text,
                                'tokens':[t+'_SR' for t in to_tokens_local(studentResponse.text)] \
                                          + matchTokens_ls})
    return responsesOut_ls
Explanation: OK, that seems to work OK. Now, let's define a function that takes a filename, and returns the list of token dictionaries:
End of explanation
extract_token_dictionaries(filename)[:2]
Explanation: We now have a function which takes a filename and returns a list of tokenised student responses and reference answers:
End of explanation
def document_frequencies(listOfTokenLists):
    # Build the dictionary of all tokens used:
    token_set=set()
    for tokenList in listOfTokenLists:
        token_set=token_set.union(set(tokenList))
        
    # Then return the document frequency counts for each token
    
    return {t:len([l for l in listOfTokenLists if t in l])
            for t in token_set}
tokenLists_ls=[x['tokens'] for x in extract_token_dictionaries(filename)]
document_frequencies(tokenLists_ls)
Explanation: So next we need to be able to build a document frequency dictionary from a list of tokenised documents.
End of explanation
from collections import Counter
def get_tfidf(tokens_ls, docFreq_dict):
    tf_dict=Counter(tokens_ls)
    return {t:tf_dict[t]/docFreq_dict[t] for t in tf_dict if t in docFreq_dict}
get_tfidf('the cat sat on the mat'.split(), {'cat':2, 'the':1})
Explanation: Next, define a function which takes a list of tokens and a document frequency dictionary, and returns a dictionary of the tf.idf values for each of the tokens in the list. Note: for this function, if a token isn't in the document frequency dictionary, then it won't be returned in the tf.idf dictionary.
We can use the collections.Counter object to get the tf values.
End of explanation
# Extract the data from the file:
tokenDictionaries_ls=extract_token_dictionaries(filename)
# Build the lists of responses:
tokenLists_ls=[x['tokens'] for x in extract_token_dictionaries(filename)]
# Build the document frequency dict
docFreq_dict=document_frequencies(tokenLists_ls)
# Create the tf.idf for each response:
tfidf_ls=[get_tfidf(tokens_ls, docFreq_dict) for tokens_ls in tokenLists_ls]
# Now, create a dataframe which is indexed by the token dictionary:
trainingText_df=pd.DataFrame(index=docFreq_dict.keys())
# Use the index of responses in the list as column headers:
for (i, tokens_ls) in enumerate(tfidf_ls):
    trainingText_df[i]=pd.Series(tokens_ls, index=trainingText_df.index)
# Finally, transpose, and replace the NaNs with 0:
trainingText_df.fillna(0).T
Explanation: Finally, we want to convert the outputs for all of the responses into a dataframe.
End of explanation
!ls semeval2013-task7/semeval2013-Task7-5way/beetle/train/Core/
Explanation: Cool, that seems to work. Now just need to do it for the complete set of files. Just use beetle/train/core for the time being.
End of explanation
import os
Explanation: Use os.walk to get the files:
End of explanation
tokenDictionaries_ls=[]
# glob would have been easier...
for (root, dirs, files) in os.walk('semeval2013-task7/semeval2013-Task7-5way/beetle/train/Core/'):
    for filename in files:
        if filename[-4:]=='.xml':
            tokenDictionaries_ls.extend(extract_token_dictionaries(os.path.join(root, filename)))
# Now we've extracted the information from all the files. We can now construct the dataframe
# in the same way as before:
# Build the lists of responses:
tokenLists_ls=[x['tokens'] for x in tokenDictionaries_ls]
# Build the document frequency dict
docFreq_dict=document_frequencies(tokenLists_ls)
# Now, create a dataframe which is indexed by the tokens
# in the token frequency dictionary:
trainingText_df=pd.DataFrame(index=docFreq_dict.keys())
# Populate the dataframe with the tf.idf for each response. Also,
# create a dictionary of the accuracy values while we're at it.
accuracy_dict={}
for (i, response_dict) in enumerate(tokenDictionaries_ls):
    trainingText_df[i]=pd.Series(get_tfidf(response_dict['tokens'], docFreq_dict), 
                                 index=trainingText_df.index)
    accuracy_dict[i]=response_dict['accuracy']
# Finally, transpose, and replace the NaNs with 0:
trainingText_df=trainingText_df.fillna(0).T
# Also, to make it easier to store in a single csv file, let's put the accuracy
# values in a column called "accuracy_txt":
trainingText_df['accuracy_txt']=pd.Series(accuracy_dict)
# And to have a final column containing a numerical equivalent of the
# accuracy_txt column (called accuracy_num ):
labels_dict={label:i for (i, label) in enumerate(set(trainingText_df['accuracy_txt']))}
trainingText_df['accuracy_num']=[labels_dict[l] for l in trainingText_df['accuracy_txt']]
trainingText_df.head()
Explanation: We can now do the same as before, but this time using all the files to construct the final dataframe. We also need a series containing the accuracy measures.
End of explanation
trainingText_df.to_csv('beetleTrainingData.csv', index=False)
Explanation: And finish by exporting to a csv file:
End of explanation
<END_TASK>