<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Simon #metoo step 1
Step2:  Data was collected using this method. It uses the Twitter API to go some days back in time, and then continuously streaming forward in real-time.
The metoo data was collected around the time when the campaign took off, but today we wouldn't be able to get data further back than about a week from now.
The video below shows the setup process for the collector.
Step3:  I had collected around 3 million metoo tweets, and they were in sqlite3 database format. We won't run the parsing process live here but the code below shows how it is done.
Step4:  For the continued analysis, I start from a csv-file that was created in this way.
Step5:  Now, let's clean the tweets, and add the clean version of the tweets as a column.
Step6:  We want to group the tweets by day, so add a column that says which day they were posted.
Step7:  Group the tweets (tweets) by day.
Step8:  View the grouping.
Step9:  Now groups is a list of tuples where element [0] is the group name, and element [1] is a pandas series of all text belonging to the group.
Step10:  The series is iterable.
Step11:  For topic modelling, we create one text document per group (day in this case).
Step12:  Now we have the topicmodelling documents.
Go back to the previous working directory.
<ASSISTANT_TASK:>
Python Code:
from IPython.display import HTML
import os
Explanation: Simon #metoo step 1
End of explanation
HTML(
<video width="320" height="240" controls>
  <source src="btf.m4v" type="video/mp4">
</video>
)
Explanation: Data was collected using this method. It uses the Twitter API to go some days back in time, and then continuously streaming forward in real-time.
The metoo data was collected around the time when the campaign took off, but today we wouldn't be able to get data further back than about a week from now.
The video below shows the setup process for the collector.
End of explanation
import sqlite3
import pandas as pd
pd.set_option('display.max_colwidth', -1)
## Read sqlite query results into a pandas DataFrame
# conn = sqlite3.connect("tweets.db")
# tweets_df = pd.read_sql_query("SELECT * from tweets", conn)
## Do some cleanupb
#tweets_df = tweets_df.replace({'\n': ' '}, regex=True) # remove linebreaks in the dataframe
#tweets_df = tweets_df.replace({'\t': ' '}, regex=True) # remove tabs in the dataframe
#tweets_df = tweets_df.replace({'\r': ' '}, regex=True) # remove carriage return in the dataframe
Explanation: I had collected around 3 million metoo tweets, and they were in sqlite3 database format. We won't run the parsing process live here but the code below shows how it is done.
End of explanation
# Read the csv into pandas
metoo_df = pd.DataFrame.from_csv("backup_metoo_171221.csv")
len(metoo_df)
metoo_df.head()
Explanation: For the continued analysis, I start from a csv-file that was created in this way.
End of explanation
newlist = []
for tweet in metoo_df['text']:
    #print(tweet)
    tweet = [word.lower() for word in tweet.split() if not word.startswith('#')]
    tweet = [word for word in tweet if not word.startswith('htt')]
    tweet = [word for word in tweet if not word.startswith('@')]
    tweet = [word for word in tweet if not '/' in word]
    tweet = [word for word in tweet if word.isalpha()]
    tweet = [word for word in tweet if not word == "rt"]
    tweet = [word.strip(".") for word in tweet]
    tweet = [word.strip() for word in tweet]
    tweet = [word for word in tweet if not word == "…"]
    tweet = " ".join(tweet)
    newlist.append(tweet)
tweets = pd.Series(newlist)
tweets
metoo_df = metoo_df.reset_index(drop=True)
metoo_df['tweets'] = tweets
metoo_df.head()
Explanation: Now, let's clean the tweets, and add the clean version of the tweets as a column.
End of explanation
daylist = [day[:10] for day in metoo_df.created_at]
daylist
days = pd.Series(daylist)
metoo_df['day'] = days
metoo_df.head()
Explanation: We want to group the tweets by day, so add a column that says which day they were posted.
End of explanation
tweets_by_day = metoo_df['tweets'].groupby(metoo_df['day'])
Explanation: Group the tweets (tweets) by day.
End of explanation
groups = list(tweets_by_day)
Explanation: View the grouping.
End of explanation
groups[0][0]
groups[0][1]
Explanation: Now groups is a list of tuples where element [0] is the group name, and element [1] is a pandas series of all text belonging to the group.
End of explanation
for i in groups[41][1]:
    print(i)
Explanation: The series is iterable.
End of explanation
os.getcwd()
try:
    os.mkdir("tmfiles")
except:
    print("Directory already exists")
os.chdir("tmfiles") # run once
os.getcwd()
for c,i in enumerate(groups):
    o = open(str(groups[c][0]) + ".txt", "w")
    for line in i[1]:
        o.write(line + "\n")
Explanation: For topic modelling, we create one text document per group (day in this case).
End of explanation
os.chdir('..')
os.getcwd()
Explanation: Now we have the topicmodelling documents.
Go back to the previous working directory.
End of explanation
<END_TASK>