<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 connecting to mongodb
Step1:  retrieving the available fields as a reference
Step2:  build a field specific stop word list
function for cleaning text
Step3:  plotting the wordcloud for abstracts and titles from various years
Step4:  plotting the number publications per year
Step5:  this shows that the database is missing entries and judging from the total number of downloaded items, there must be around 30000 papers missing 
plotting the relative appearance of a few terms
Step6:  plotting the world map of author affiliations
there is no affiliation in the parsed data
Step7:  using LDA to map out the topics
<ASSISTANT_TASK:>
Python Code:
client = MongoClient('localhost:27017')
db = client.arXivDB
db.arXivfeeds.count()
Explanation: connecting to mongodb
End of explanation
print(db.arXivfeeds.find_one().keys())
for item in db.arXivfeeds.find({'published_parsed': 2016}).sort('_id', pymongo.DESCENDING).limit(5):
    print(item['title'])
#db.arXivfeeds.delete_many({})
Explanation: retrieving the available fields as a reference
End of explanation
def cleaner(doc, stem=False):
    '''Function to clean the text data and prep for further analysis'''
    doc = doc.lower() # turn text to lowercase
    stops = set(stopwords.words("english"))       # Creating a set of Stopwords
    p_stemmer = PorterStemmer()                   # Creating the stemmer model
    doc = re.sub(r"quantum", '', doc)           # removing the word quantum (duh)
    doc = re.sub(r"physics", '', doc)           # removing the word physics (duh)
    doc = re.sub(r"state", '', doc)           # removing the word state (duh)
    doc = re.sub(r'\$.*?\$', 'latexinlineformula', doc) # replacing latex inline formula
    doc = re.sub(r'\\n', ' ', doc) # removing new line character
    doc = re.sub(r'\\\\\"', '', doc)             # removing german double dotted letters
    doc = re.sub(r"</?\w+[^>]*>", '', doc)      # removing html tags
    doc = re.sub("[^a-zA-Z]", ' ', doc)    # removing anythin other alpha-numerical char's and @ and !
    doc = doc.split()                          # Splits the data into individual words 
    doc = [w for w in doc if not w in stops and len(w) > 3]   # Removes stopwords and short length words
    if stem:
        doc = [p_stemmer.stem(i) for i in doc]     # Stemming (reducing words to their root)
    if not len(doc):                            # dealing with comments that are all emojis, stop words or other languages
        doc = ['emptystring']
    # print('text cleaning done!')
    return ' '.join(doc)
cleaner(db.arXivfeeds.find_one({'published_parsed': 2016})['summary'])
Explanation: build a field specific stop word list
function for cleaning text
End of explanation
def plot_abstract_and_title_wordcloud(arXivfeed_query_result):
    
    arXivfeed_2015_text = cleaner(' '.join([' '.join(list(d.values())) for d in arXivfeed_query_result]))
    # Generate a word cloud image
    wordcloud_arXivfeed_2015 = WordCloud().generate(arXivfeed_2015_text)
    # Display the generated image:
    plt.imshow(wordcloud_arXivfeed_2015)
    plt.axis("off")
plot_abstract_and_title_wordcloud(list(db.arXivfeeds.find({'published_parsed': 1995}, {'_id':0,'title':1})))
plot_abstract_and_title_wordcloud(list(db.arXivfeeds.find({'published_parsed': 2002}, {'_id':0,'title':1})))
plot_abstract_and_title_wordcloud(list(db.arXivfeeds.find({'published_parsed': 2015}, {'_id':0,'title':1})))
Explanation: plotting the wordcloud for abstracts and titles from various years
End of explanation
years = range(1994,2016,1)
num_publications_per_year = [db.arXivfeeds.find({'published_parsed': y}).count() for y in years]
plt.plot(years, num_publications_per_year)
Explanation: plotting the number publications per year
End of explanation
pattern1 = r'[Pp]hoton\w*'
pattern2 = r'[Oo]ptic\w*'
set(re.findall(pattern2, 
               ' '.join([' '.join(list(d.values())) for d in db.arXivfeeds.find({}, {'_id':0,'summary':1})])))
num_ph_papers = np.zeros(len(years))
for i, y in enumerate(years):
    num_ph_papers[i] = db.arXivfeeds.find({'$and':[{'published_parsed': y},
                        {'$or':[
                            {'summary': {'$regex': pattern1}},
                            {'title': {'$regex': pattern1}},
                            {'summary': {'$regex': pattern2}},
                            {'title': {'$regex': pattern2}}
                        ]}
                        ]}).count()
plt.plot(years, num_ph_papers/num_publications_per_year)
Explanation: this shows that the database is missing entries and judging from the total number of downloaded items, there must be around 30000 papers missing 
plotting the relative appearance of a few terms
End of explanation
list(db.arXivfeeds.find({'published_parsed': 2016}).limit(1))[0]
Explanation: plotting the world map of author affiliations
there is no affiliation in the parsed data
End of explanation
import nltk
import gensim
import pyLDAvis
import pyLDAvis.gensim
documents = [cleaner(d['summary']) for d in db.arXivfeeds.find({'published_parsed': 2010}, {'_id':0, 'summary':1})]
# documents = [cleaner(d['summary']) for d in db.arXivfeeds.find({}, {'_id':0, 'summary':1})]
train_set = []
for j in range(len(documents)):
    train_set.append(nltk.word_tokenize(documents[j]))
dic = gensim.corpora.Dictionary(train_set)
print(len(dic))
dic.filter_extremes(no_below=20, no_above=0.1)
print(len(dic))
corpus = [dic.doc2bow(text) for text in train_set]  # transform every token into BOW
tfidf = gensim.models.TfidfModel(corpus)
corpus_tfidf = tfidf[corpus]
lda = gensim.models.LdaModel(corpus_tfidf, id2word = dic, num_topics = 10, iterations=20, passes = 10)
corpus_lda = lda[corpus_tfidf]
vis_data = pyLDAvis.gensim.prepare(lda, corpus, dic)
pyLDAvis.display(vis_data)
Explanation: using LDA to map out the topics
End of explanation
<END_TASK>