<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Normalizing text
Step1:  Normalizing columns
Step2:  Answers in questions
Step3:  Only 0.6% of the answers appear in the questions itself. Out of this 0.6%, a sample of the questions shows that they are all multiple choice questions, which concludes that it is very unlikely that the answer will be in the question itself.
Recycled questions
Step4:  Low value vs high value questions
<ASSISTANT_TASK:>
Python Code:
import string
def norm_words(words):
    words = words.lower().translate(None, string.punctuation)
    return words
jeopardy["clean_question"] = jeopardy["Question"].apply(norm_words)
jeopardy["clean_answer"] = jeopardy["Answer"].apply(norm_words)
jeopardy.head()
Explanation: Normalizing text
End of explanation
def norm_value(value):
    try:
        value = int(value.translate(None, string.punctuation))
    except:
        value = 0
    return value
jeopardy["clean_value"] = jeopardy["Value"].apply(norm_value)
jeopardy["Air Date"] = pd.to_datetime(jeopardy["Air Date"])
print(jeopardy.dtypes)
jeopardy.head()
Explanation: Normalizing columns
End of explanation
def ans_in_q(row):
    match_count = 0
    split_answer = row["clean_answer"].split(" ")
    split_question = row["clean_question"].split(" ")
    
    try:
        split_answer.remove("the")
    except:
        pass
    
    if len(split_answer) == 0:
        return 0
    else:
        for word in split_answer:
            if word in split_question:
                match_count += 1
        return match_count / len(split_answer)
jeopardy["answer_in_question"] = jeopardy.apply(ans_in_q, axis=1)
print(jeopardy["answer_in_question"].mean())
jeopardy[jeopardy["answer_in_question"] > 0].head()
jeopardy[(jeopardy["answer_in_question"] > 0) & (jeopardy["clean_question"].apply(string.split).apply(len) > 6)].head()
Explanation: Answers in questions
End of explanation
jeopardy = jeopardy.sort_values(by="Air Date")
question_overlap = []
terms_used = set()
for index, row in jeopardy.iterrows():
    match_count = 0
    split_question = row["clean_question"].split(" ")
    for word in split_question:
        if len(word) < 6:
            split_question.remove(word)
    for word in split_question:
        if word in terms_used:
            match_count += 1
        terms_used.add(word)
    if len(split_question) > 0:
        match_count /= float(len(split_question))
    question_overlap.append(match_count)
jeopardy["question_overlap"] = question_overlap
print(jeopardy["question_overlap"].mean())
jeopardy.tail()
Explanation: Only 0.6% of the answers appear in the questions itself. Out of this 0.6%, a sample of the questions shows that they are all multiple choice questions, which concludes that it is very unlikely that the answer will be in the question itself.
Recycled questions
End of explanation
def value(row):
    if row["clean_value"] > 800:
        value = 1
    else:
        value = 0
    return value
jeopardy["high_value"] = jeopardy.apply(value, axis=1)
jeopardy.head()
Explanation: Low value vs high value questions
End of explanation
<END_TASK>