<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Home Depot Product Search Relevance
The challenge is to predict a relevance score for the provided combinations of search terms and products. To create the ground truth labels, Home Depot has crowdsourced the search/product pairs to multiple human raters.
LabGraph Create
This notebook uses the LabGraph create machine learning iPython module. You need a personal licence to run this code.
Step1:  Load data from CSV files
Step2:  Data merging
Step3:  Let's explore some data
Let's examine 3 different queries and products
Step4:  'angle bracket' search term is not contained in the body. 'angle' would be after stemming however 'bracket' is not.
Step5:  only 'wood' is present from search term
Step6:  'sheer' and 'courtain' are present and that's all
How many search terms are not present in description and title for ranked 3 documents
Ranked 3 documents are the most relevents searches, but how many search queries doesn't include the searched term in the description and the title
Step7:  Stemming
Step8:  TF-IDF with linear regression
<ASSISTANT_TASK:>
Python Code:
import graphlab as gl
from nltk.stem import *
Explanation: Home Depot Product Search Relevance
The challenge is to predict a relevance score for the provided combinations of search terms and products. To create the ground truth labels, Home Depot has crowdsourced the search/product pairs to multiple human raters.
LabGraph Create
This notebook uses the LabGraph create machine learning iPython module. You need a personal licence to run this code.
End of explanation
train = gl.SFrame.read_csv("../data/train.csv")
test = gl.SFrame.read_csv("../data/test.csv")
desc = gl.SFrame.read_csv("../data/product_descriptions.csv")
Explanation: Load data from CSV files
End of explanation
# merge train with description
train = train.join(desc, on = 'product_uid', how = 'left')
# merge test with description
test = test.join(desc, on = 'product_uid', how = 'left')
Explanation: Data merging
End of explanation
first_doc = train[0]
first_doc
Explanation: Let's explore some data
Let's examine 3 different queries and products:
* first from the training set
* somewhere in the moddle in the training set
* the last one from the training set
End of explanation
middle_doc = train[37033]
middle_doc
Explanation: 'angle bracket' search term is not contained in the body. 'angle' would be after stemming however 'bracket' is not.
End of explanation
last_doc = train[-1]
last_doc
Explanation: only 'wood' is present from search term
End of explanation
train['search_term_word_count'] = gl.text_analytics.count_words(train['search_term'])
ranked3doc = train[train['relevance'] == 3]
print ranked3doc.head()
len(ranked3doc)
words_search = gl.text_analytics.tokenize(ranked3doc['search_term'], to_lower = True)
words_description = gl.text_analytics.tokenize(ranked3doc['product_description'], to_lower = True)
words_title = gl.text_analytics.tokenize(ranked3doc['product_title'], to_lower = True)
wordsdiff_desc = []
wordsdiff_title = []
puid = []
search_term = []
ws_count = []
ws_count_used_desc = []
ws_count_used_title = []
for item in xrange(len(ranked3doc)):
    ws = words_search[item]
    pd = words_description[item]
    pt = words_title[item]
    diff = set(ws) - set(pd)
    if diff is None:
        diff = 0
    wordsdiff_desc.append(diff)
    
    diff2 = set(ws) - set(pt)
    if diff2 is None:
        diff2 = 0
    wordsdiff_title.append(diff2)
    
    puid.append(ranked3doc[item]['product_uid'])
    search_term.append(ranked3doc[item]['search_term'])
    ws_count.append(len(ws))
    ws_count_used_desc.append(len(ws) - len(diff))
    ws_count_used_title.append(len(ws) - len(diff2))
    
differences = gl.SFrame({"puid" : puid,
                         "search term": search_term,
                         "diff desc" : wordsdiff_desc,
                         "diff title" : wordsdiff_title,
                         "ws count" : ws_count, 
                         "ws count used desc" : ws_count_used_desc,
                         "ws count used title" : ws_count_used_title})
differences.sort(['ws count used desc', 'ws count used title'])
print "No terms used in description : " + str(len(differences[differences['ws count used desc'] == 0]))
print "No terms used in title : " + str(len(differences[differences['ws count used title'] == 0]))
print "No terms used in description and title : " + str(len(differences[(differences['ws count used desc'] == 0) & 
                                                                        (differences['ws count used title'] == 0)]))
import matplotlib.pyplot as plt
%matplotlib inline
Explanation: 'sheer' and 'courtain' are present and that's all
How many search terms are not present in description and title for ranked 3 documents
Ranked 3 documents are the most relevents searches, but how many search queries doesn't include the searched term in the description and the title
End of explanation
#stemmer = SnowballStemmer("english")
stemmer = PorterStemmer()
def stem(word):
    singles = [stemmer.stem(plural) for plural in unicode(word, errors='replace').split()]
    text = ' '.join(singles)
    return text
print "Starting stemming train search term..."
stemmed = train['search_term'].apply(stem)
train['stem_search_term'] = stemmed
print "Starting stemming train product description..."
stemmed = train['product_description'].apply(stem)
train['stem_product_description'] = stemmed
print "Starting stemming train product title..."
stemmed = train['product_title'].apply(stem)
train['stem_product_title'] = stemmed
print "Starting stemming test search term..."
stemmed = test['search_term'].apply(stem)
test['stem_search_term'] = stemmed
print "Starting stemming test product description..."
stemmed = test['product_description'].apply(stem)
test['stem_product_description'] = stemmed
print "Starting stemming test product title..."
stemmed = test['product_title'].apply(stem)
test['stem_product_title'] = stemmed
Explanation: Stemming
End of explanation
train['search_term_word_count'] = gl.text_analytics.count_words(train['stem_search_term'])
train_search_tfidf = gl.text_analytics.tf_idf(train['search_term_word_count'])
train['search_tfidf'] = train_search_tfidf
train['product_desc_word_count'] = gl.text_analytics.count_words(train['stem_product_description'])
train_desc_tfidf = gl.text_analytics.tf_idf(train['product_desc_word_count'])
train['desc_tfidf'] = train_desc_tfidf
train['product_title_word_count'] = gl.text_analytics.count_words(train['stem_product_title'])
train_title_tfidf = gl.text_analytics.tf_idf(train['product_title_word_count'])
train['title_tfidf'] = train_title_tfidf
train['distance_desc'] = train.apply(lambda x: gl.distances.cosine(x['search_tfidf'],x['desc_tfidf']))
#train['distance_desc_sqrt'] = train['distance_desc'] ** 2
train['distance_title'] = train.apply(lambda x: gl.distances.cosine(x['search_tfidf'],x['title_tfidf']))
#train['distance_title_sqrt'] = train['distance_title'] ** 3
model1 = gl.linear_regression.create(train, target = 'relevance', 
                                         features = ['distance_desc', 'distance_title'], 
                                         validation_set = None)
# model1 = gl.linear_regression.create(train, target = 'relevance', 
#                                         features = ['distance_desc', 'distance_desc_sqrt', 'distance_title', 'distance_title_sqrt'], 
#                                         validation_set = None)
#let's take a look at the weights before we plot
model1.get("coefficients")
test['search_term_word_count'] = gl.text_analytics.count_words(test['stem_search_term'])
test_search_tfidf = gl.text_analytics.tf_idf(test['search_term_word_count'])
test['search_tfidf'] = test_search_tfidf
test['product_desc_word_count'] = gl.text_analytics.count_words(test['stem_product_description'])
test_desc_tfidf = gl.text_analytics.tf_idf(test['product_desc_word_count'])
test['desc_tfidf'] = test_desc_tfidf
test['product_title_word_count'] = gl.text_analytics.count_words(test['stem_product_title'])
test_title_tfidf = gl.text_analytics.tf_idf(test['product_title_word_count'])
test['title_tfidf'] = test_title_tfidf
test['distance_desc'] = test.apply(lambda x: gl.distances.cosine(x['search_tfidf'],x['desc_tfidf']))
#test['distance_desc_sqrt'] = test['distance_desc'] ** 2
test['distance_title'] = test.apply(lambda x: gl.distances.cosine(x['search_tfidf'],x['title_tfidf']))
#test['distance_title_sqrt'] = test['distance_title'] ** 3
'''
predictions_test = model1.predict(test)
test_errors = predictions_test - test['relevance']
RSS_test = sum(test_errors * test_errors)
print RSS_test
'''
predictions_test = model1.predict(test)
predictions_test
submission = gl.SFrame(test['id'])
submission.add_column(predictions_test)
submission.rename({'X1': 'id', 'X2':'relevance'})
submission['relevance'] = submission.apply(lambda x: 3.0 if x['relevance'] > 3.0 else x['relevance'])
submission['relevance'] = submission.apply(lambda x: 1.0 if x['relevance'] < 1.0 else x['relevance'])
submission['relevance'] = submission.apply(lambda x: str(x['relevance']))
submission.export_csv('../data/submission2.csv', quote_level = 3)
#gl.canvas.set_target('ipynb')
Explanation: TF-IDF with linear regression
End of explanation
<END_TASK>