<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Text Analysis
Notebook by
Step1:  Load
Step2:  Account for NaN in column name.
Random Sample
Step3:  Transform
This code
Step4:  Bag of Words
Tokenize
Step5:  Stop Words
Removing English stopwords from NLTK.
Step6:  This code
Step7:  To DF
Step8:  Words to Features
Step9:  LDA
Step10:  Topics
Merge the topic assignments when number of topics is 10.
<ASSISTANT_TASK:>
Python Code:
import re
import random
#import lda
import csv
import numpy as np
import pandas as pd
from collections import Counter
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from sklearn.feature_extraction.text import CountVectorizer
Explanation: Text Analysis
Notebook by: Juan Shishido
In this notebook, I'll start cleaning the text columns and, more importantly, thinking about how to classify and group the data within them. Consider using n-grams for word occurence.
Imports
End of explanation
df = pd.read_csv('../../data/cleaned/UCB_dept_merge.csv')
df['product_line'] = df['supplier_name'] + ' ' + \
                     df['product_description'] + ' ' + \
                     df['manufacturer']
Explanation: Load
End of explanation
random.seed(8675309)
rows = random.sample(df.index, 40000)
df = df.ix[rows].reset_index()
df.head()
Explanation: Account for NaN in column name.
Random Sample: 40,000
End of explanation
cols = ['supplier_name', 'item_type', 'product_description', 'manufacturer', 
        'buyer__first_name', 'buyer__last_name', 'department_name', 'product_line']
for col in cols:
    df[col] = df[col].replace(np.nan, '' , regex=True)                                      \
                     .apply(lambda x: x.lower())                                            \
                     .apply(lambda x: re.sub('(http\S*|www\S*)', '', x))                    \
                     .apply(lambda x: re.sub('((?<=\D)/|/(?=\D))', ' ', x))                 \
                     .apply(lambda x: re.sub('[^A-Za-z0-9.%\/]+', ' ', x))                  \
                     .apply(lambda x: re.sub('\.+', '', x))                                 \
                     .apply(lambda x: re.sub('(?<=\s)\w(?=\s)|(?<=\s)\d(?=\s)', '', x))     \
                     .apply(lambda x: re.sub('\s+', ' ', x).strip())
df.head()
Explanation: Transform
This code:
removes NaNs
converts all to lowercase
removes URLs
replaces forward slashes not associated with fractions with a single space
replaces characters that are not alphanumeric, periods, percent signs, or forward slashes with a single space
removes multiple periods
removes individual letters (with spaces on either side)
removes multiple whitespace
In a later step, numbers not associated with a percent sign are removed.
End of explanation
tokenized_pd = [word_tokenize(line) for line in df.product_line]
Explanation: Bag of Words
Tokenize
End of explanation
stop_words = stopwords.words('english') + \
             [u'ea', u'per', u'item', u'description', u'quote', u'pk', u'pack',
              'give', 'something', 'inc', 'corporation', 'quantity', 'back',
              'products', 'co', 'officemax', 'unit', 'corp']
Explanation: Stop Words
Removing English stopwords from NLTK.
End of explanation
tokenized_pd_clean = []
for entry in tokenized_pd:
    entry_list = []
    for word in entry:
        if ((not word in stop_words) and \
        (not unicode(word).isnumeric()) and \
        (not len(word) <= 1)):
            entry_list.append(word)
    tokenized_pd_clean.append(entry_list)
Explanation: This code:
removes English stop words
removes numbers (excludes numbers with non-numeric characters attached)
removes single characters
End of explanation
df['tokenized_pd_clean'] = tokenized_pd_clean
Explanation: To DF
End of explanation
pd_list_clean = []
for item in tokenized_pd_clean:
    pd_list_clean.append(' '.join(item))
vectorizer = CountVectorizer(analyzer = "word", 
                             tokenizer = None, 
                             preprocessor = None, 
                             stop_words = None)
word_features = vectorizer.fit_transform(pd_list_clean).toarray()
word_features.shape
word_features[0:5,:]
vocab = vectorizer.get_feature_names()
print vocab[:15]
vocab_map = vectorizer.vocabulary_
Explanation: Words to Features
End of explanation
X = word_features
model = lda.LDA(n_topics=15, n_iter=1500, random_state=8675309)
model.fit(X)
topic_word = model.topic_word_
n_top_words = 21
with open('../../results/topic_definitions.csv', 'wb') as to_:
    writer = csv.writer(to_, delimiter=',', quotechar='\"')
    doc_topic = model.doc_topic_
    for i, topic_dist in enumerate(topic_word):
        topic_words = np.array(vocab)[np.argsort(topic_dist)][:-n_top_words:-1]
        writer.writerow([i, ' '.join(topic_words)])
with open('../../results/pd_topics.csv', 'wb') as to_:
    writer = csv.writer(to_, delimiter=',', quotechar='\"')
    doc_topic = model.doc_topic_
    for i in range(len(tokenized_pd_clean)):
        writer.writerow([tokenized_pd_clean[i], doc_topic[i].argmax()])
words = [w.strip().split(' ') for w in pd_list_clean]
word_list = [i for word in words for i in word]
word_counts = Counter(word_list)
top_100_words = word_counts.most_common(100)
for word in top_100_words:
    print word
Explanation: LDA
End of explanation
topics = pd.read_csv('../../results/pd_topics_10.csv', header=None)
topics.columns = ['tpc', 'topic']
df['tpc'] = topics.tpc
df['topic'] = topics.topic
depts = pd.DataFrame({'count' : df.groupby('department_name')['department_name'].count()}).reset_index()
depts.sort('count', ascending=False, inplace=True)
top15 = depts['department_name'][:25].tolist()
df_top15 = df[df.department_name.isin(top15)]
df_top15 = df_top15[df_top15['product_line'] != '']
topics_by_dept = pd.DataFrame({'count' : df_top15.groupby(['department_name', 'topic'])['topic'].count()}).reset_index()
topic_def = pd.read_csv('../../results/topics_definitions_10.csv', header=None)
topic_def.columns = ['topic', 'words']
topic_def['words'] = topic_def['words'].apply(lambda x: ', '.join(x.split()[:10]))
df_top15_final = pd.merge(topics_by_dept, topic_def, on='topic')
df_top15_final.to_csv('../../results/topic_count_10.csv', index=False)
Explanation: Topics
Merge the topic assignments when number of topics is 10.
End of explanation
<END_TASK>