<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 In this notebook, we will train an Latend Dirichlet Allocation (LDA) model on tweets to learn a set of words which commonly appear together, hopefully corresponding to a topic.
We will apply the LDA training on the whole corpus of our tweets and extract 10 topics. Additionally, we will visualize the results using the pyLDAvis library.
Following, we will take these results to a different notebook for analysis. There, we will assign a topic distribution on each tweet given the words used in it and we will sum the topic distributions of all tweets corresponding to a state to conclude to the topic distribution per state.
Step1:  LOAD data from Mongo
Step2:  Preproccessing
Step3:  save/load the model
Step4:  efforts with pyLDAvis (visualize the LDA topics)
Step5:  Interactive version available here
Also available as an html document in this git repository (LDA_topics.html)
<ASSISTANT_TASK:>
Python Code:
from pymongo import MongoClient
import json
client = MongoClient()
db = client.Twitter
import pandas as pd
import time
import re
from nltk.tokenize import RegexpTokenizer
import HTMLParser # In Python 3.4+ import html 
import nltk
from nltk.corpus import stopwords
Explanation: In this notebook, we will train an Latend Dirichlet Allocation (LDA) model on tweets to learn a set of words which commonly appear together, hopefully corresponding to a topic.
We will apply the LDA training on the whole corpus of our tweets and extract 10 topics. Additionally, we will visualize the results using the pyLDAvis library.
Following, we will take these results to a different notebook for analysis. There, we will assign a topic distribution on each tweet given the words used in it and we will sum the topic distributions of all tweets corresponding to a state to conclude to the topic distribution per state.
End of explanation
start_time = time.time()
#we are filtering out tweets of different languages and outside of the US
filter_query = { 
    "$and":[ {"place.country_code":"US"}, { "lang": "en" } ]
    }
#we are keeping only our fields of interest
columns_query = {
    'text':1,
    'entities.hashtags':1,
    'entities.user_mentions':1,
    'place.full_name':1,
    'place.bounding_box':1
}
tweets = pd.DataFrame(list(db.tweets.find(
    filter_query, 
    columns_query
    )#.limit()
                               )
                          )
elapsed_time = time.time() - start_time
print elapsed_time
Explanation: LOAD data from Mongo
End of explanation
#parse state variable
tweets['state'] = map(lambda place_dict: place_dict['full_name'][-2:] ,tweets['place'])
tweets['state'].value_counts().head()
# #for one state only
# state = 'CA'
# tweets = tweets[tweets['state']==state]
len(tweets)
def Clean(unescaped_tweet):
    '''This function takes a tweet as input and returns a tokenizing list.'''
    
    tokenizer = RegexpTokenizer(r'\w+')
    cleaned_tweet_tokens = tokenizer.tokenize(unescaped_tweet.lower())
    
    return cleaned_tweet_tokens
start_time = time.time() #Starts time
tweets['text'] = tweets['text'].apply(lambda tweet: re.sub(r"http\S+", "", tweet))
#########################################################
def trump_mention(tweet):
    trump_count = 0
    if ('trump' in tweet.lower()) or ('donald'  in tweet.lower()):
        
        return True
        
    return False
tweets['Trump'] = tweets['text'].apply(lambda tweet: trump_mention(tweet))
##############################################################
#tweet mentions --->@
#tweet hashtags --->#
#create two column with the the hashtags and the mentions
tweets['mentions'] = tweets['text'].apply(lambda tweet: re.findall(r'\@\w+',tweet))
tweets['hashtags'] = tweets['text'].apply(lambda tweet: re.findall(r'\#\w+',tweet))
#remove hashtags and mentions
tweets['text'] = tweets['text'].apply(lambda tweet: re.sub(r"\@\w+" , "", tweet))
tweets['text'] = tweets['text'].apply(lambda tweet: re.sub(r"\#\w+" , "", tweet))
#remove the numbers from the text
tweets['text'] =tweets['text'].apply(lambda tweet: ''.join([i for i in tweet if not i.isdigit()]))
trump_count = 0
clinton_count =0
#remove the names and surnames of the two candidates
tweets['text'] =tweets['text'].apply(lambda tweet: re.sub(r"Trump" , "", tweet))
tweets['text'] =tweets['text'].apply(lambda tweet: re.sub(r"Clinton" , "", tweet))
tweets['text'] =tweets['text'].apply(lambda tweet: re.sub(r"Donald" , "", tweet))
tweets['text'] =tweets['text'].apply(lambda tweet: re.sub(r"Hillary" , "", tweet))
tweets['text'] =tweets['text'].apply(lambda tweet: re.sub(r"USA" , "", tweet))
tweets['text'] =tweets['text'].apply(lambda tweet: re.sub(r"amp" , "", tweet))
#tokenize the text and add an extra column
tweets['token'] = tweets['text'].apply(lambda tweet: Clean(tweet))
tweets['token'] = tweets['token'].apply(lambda x: list(set(x)-set(stopwords.words('english'))))
elapsed_time = time.time() - start_time #time ends
print elapsed_time
tweets.head()
tweets.head()
#test['tags'] = map(lambda tweet: map(lambda tweet: tweet['text'] , tweet['entities']['hashtags']) if tweet['entities']['hashtags'] != None else None, raw_tweet[:100])
#tweets['text'][9]
doc_complete = tweets['token'].tolist()
doc_complete[:2]
import gensim
import pickle
import gensim
from gensim import corpora
# Creating the term dictionary of our courpus, where every unique term is assigned an index
dictionary = corpora.Dictionary(doc_complete)
pickle.dump(dictionary, open( 'dictionary2.pickle', "wb" ) )
# Converting list of documents (corpus) into Document Term Matrix using dictionary prepared above.
doc_term_matrix = [dictionary.doc2bow(doc) for doc in doc_complete]
pickle.dump(doc_term_matrix, open( 'doc_term_matrix.pickle', "wb" ) )
Lda = gensim.models.ldamulticore.LdaMulticore
nr_topics = 10
nr_passes = 100
start_time = time.time()
# Creating the object for LDA model using gensim library
# Running and Trainign LDA model on the document term matrix.
ldamodel = Lda(doc_term_matrix, num_topics=nr_topics, id2word = dictionary, passes=nr_passes)
elapsed_time = time.time() - start_time
print 'Topic modelling for', nr_topics,'topics,', nr_passes,'passes,',len(tweets),'tweets:','\ncomplete in',elapsed_time/60.,'minutes'
# Runtimes:
# Florida (~4K) ~ 16 min on 10 topics, 300 passes
# CA (57K) - 48 min on 10 topics 300 passes
# can we do it on the whole data -> take the topics and classify each tweet within them.
# then we have discrete sets with topics and words weights in each topic.
# so then isn't a tweet represented by the appropriate values?
# Print 2 topics and describe then with 4 words.
topics = ldamodel.print_topics(num_topics=nr_topics, num_words=50)
i=0
for topic in topics:
    print topic
    print ""
    i+=1
Explanation: Preproccessing
End of explanation
import pickle
nr_topics = 10
nr_passes = 100
state = 'allstates'
name = "trained models/lda/%s_%itopics_%ipasses.pickle"%(state,nr_topics,nr_passes)
print "Procceed to save model in:", name
pickle.dump(ldamodel, open( name, "wb" ) )
#load
ldamodel = pickle.load(open(name,'rb'))
Explanation: save/load the model
End of explanation
import time
import pyLDAvis.gensim
pyLDAvis.enable_notebook()
#load the LDA results (model, dictionary and corpus)
start_time = time.time()
ldamodel = pickle.load(open('trained models/lda/allstates_10topics_100passes.pickle'))
dictandcorpus = pickle.load(open('trained models/lda/Dictionary.pickle'))
c = dictandcorpus[1]
d = dictandcorpus[0]
del dictandcorpus
elapsed_time = time.time() - start_time
print elapsed_time
data = pyLDAvis.gensim.prepare(ldamodel, c, d)
data
Explanation: efforts with pyLDAvis (visualize the LDA topics)
End of explanation
#save results as an html file
pyLDAvis.save_html(data, open('LDA topics.html','wb'))
Explanation: Interactive version available here
Also available as an html document in this git repository (LDA_topics.html)
End of explanation
<END_TASK>