<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Loading and processing datasets of all bot-bot reverts for all languages
This dataset loads and parses the reverted_bot2bot datasets for seven languages, which were created by processing the Wikipedia revision history dumps by the scripts called in the Makefile in the root directory of the repository. This is the first script that you can run based entirely off the files in this GitHub repository. This generates
Step1:  Data processing
Initial datasets of bot-bot reverts are stored in datasets/reverted_bot2bot/ as TSV files compressed in bzip2 format for each language.
We first put them in a dict of dataframes {language
Step2:  See what we've got
Step3:  Combining into one tidy dataframe for all languages
Step4:  Number of bot-bot reverts, up to 2017-04-20
Step6:  Namespace type
Step7:  Datetime parsing
Step8:  Filter all datasets to the same time bounds
Step9:  Counts per year
Step10:  Counts per year
Step12:  Other processing and metadata
Comment parsing
Step15:  Functions for calculating botpair and botpair_sorted
Step16:  Reverts per page per botpair
This analysis is also replicated in R in reverts_per_page_R.ipynb
Step17:  Check time to revert for negatives
Step18:  Final data format
Step19:  Output to file
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
import glob
import datetime
%matplotlib inline
pd.set_option("display.max_columns",100)
start = datetime.datetime.now()
Explanation: Loading and processing datasets of all bot-bot reverts for all languages
This dataset loads and parses the reverted_bot2bot datasets for seven languages, which were created by processing the Wikipedia revision history dumps by the scripts called in the Makefile in the root directory of the repository. This is the first script that you can run based entirely off the files in this GitHub repository. This generates:
- /datasets/parsed_dataframes/df_all_2016.pickle
- /datasets/parsed_dataframes/df_all_2016.pickle.xz
These datasets are used for the analyses in section 5 (5-*.ipynb) and as the basis of the comment parsing analysis in section 7 and 8.
This entire notebook can be run from the beginning with Kernel -> Restart & Run All in the menu bar. On a laptop running a Core i5-2540M processor, it takes about 5 minutes to run, then another 5 minutes to compress to xz.
End of explanation
!ls -lah ../../datasets/reverted_bot2bot/*.bz2
!bunzip2 -kf ../../datasets/reverted_bot2bot/*.bz2
!ls -lah ../../datasets/reverted_bot2bot/*.tsv
glob.glob("../../datasets/reverted_bot2bot/??wiki_20170420.tsv")
df_dict = {}
for filename in glob.glob("../../datasets/reverted_bot2bot/??wiki_20170420.tsv"):
    lang_code = filename[32:34]
    df_dict[lang_code] = pd.read_csv(filename, sep="\t")
    df_dict[lang_code] = df_dict[lang_code].drop_duplicates()
Explanation: Data processing
Initial datasets of bot-bot reverts are stored in datasets/reverted_bot2bot/ as TSV files compressed in bzip2 format for each language.
We first put them in a dict of dataframes {language:dataframe} then make it into a nice and tidy unified dataframe.
End of explanation
for lang, lang_df in df_dict.items():
    print(lang, len(lang_df))
df_dict['en'][0:2].transpose()
Explanation: See what we've got
End of explanation
df_all = df_dict['en'].copy()
df_all = df_all.drop(df_all.index, axis=0)
for lang, lang_df in df_dict.items():
    lang_df['language'] = lang
    df_all = pd.concat([df_all, lang_df])
df_all['language'].value_counts()
Explanation: Combining into one tidy dataframe for all languages
End of explanation
len(df_all)
Explanation: Number of bot-bot reverts, up to 2017-04-20
End of explanation
def namespace_type(item):
    
    Classifies namespace type. To be used with df.apply() on ['page_namespace']
        
    if int(item) == 0:
        return 'article'
    elif int(item) == 14:
        return 'category'
    elif int(item) % 2 == 1:
        return 'other talk'
    else:
        return 'other page'
df_all['namespace_type'] = df_all['page_namespace'].apply(namespace_type)
df_all['namespace_type'].value_counts()
Explanation: Namespace type
End of explanation
def get_year(timestamp):
    return timestamp.year
df_all['reverting_timestamp_dt'] = pd.to_datetime(df_all['reverting_timestamp'], format="%Y%m%d%H%M%S")
df_all['reverted_timestamp_dt'] = pd.to_datetime(df_all['rev_timestamp'], format="%Y%m%d%H%M%S")
df_all = df_all.set_index('reverting_timestamp_dt')
df_all['reverting_timestamp_dt'] = pd.to_datetime(df_all['reverting_timestamp'], format="%Y%m%d%H%M%S")
df_all['time_to_revert'] = df_all['reverting_timestamp_dt']-df_all['reverted_timestamp_dt']
df_all['time_to_revert_hrs'] = df_all['time_to_revert'].astype('timedelta64[s]')/(60*60)
df_all['time_to_revert_days'] = df_all['time_to_revert'].astype('timedelta64[s]')/(60*60*24)
df_all['reverting_year'] = df_all['reverting_timestamp_dt'].apply(get_year)
df_all['time_to_revert_days_log10'] = df_all['time_to_revert_days'].apply(np.log10)
df_all['time_to_revert_hrs_log10'] = df_all['time_to_revert_hrs'].apply(np.log10)
Explanation: Datetime parsing
End of explanation
df_all = df_all.loc["2001-01-01":"2016-12-31"]
Explanation: Filter all datasets to the same time bounds: 2001 to 2016
End of explanation
df_all[df_all['language']=='en'].reverting_year.value_counts().sort_index()
Explanation: Counts per year: en
End of explanation
df_all.reverting_year.value_counts().sort_index()
Explanation: Counts per year: all
End of explanation
# by http://stackoverflow.com/questions/14596884/remove-text-between-and-in-python
def remove_brackets(test_str):
    
    Takes a string and returns that string with text in brackets and parentheses removed
    
    
    test_str = str(test_str)
    ret = ''
    skip1c = 0
    skip2c = 0
    for i in test_str:
        if i == '[':
            skip1c += 1
        elif i == '(':
            skip2c += 1
        elif i == ']' and skip1c > 0:
            skip1c -= 1
        elif i == ')'and skip2c > 0:
            skip2c -= 1
        elif skip1c == 0 and skip2c == 0:
            ret += i
            
    return " ".join(ret.split())
df_all['reverting_comment_nobracket'] = df_all['reverting_comment'].apply(remove_brackets)
Explanation: Other processing and metadata
Comment parsing: removing text in brackets/parens
Function for removing text within square brackets or parentheses, which is useful for aggregating comment messages.
End of explanation
def concat_botpair(row):
    
    Concatenate the reverting and reverted user names. To be used with df.apply()
    on the entire row
    
    return str(row['reverting_user_text']) + " rv " + str(row['rev_user_text'])
def sorted_botpair(row):
    
    Returns a sorted list of bot pairs (reverted and reverting). To be used with
    df.apply() on the entire row. list.sort() is  locale dependent, but it doesn't
    matter because all we need is a consistent way of uniquely sorting.
    
    
    return str(sorted([row['reverting_user_text'], row['rev_user_text']]))
df_all['botpair'] = df_all.apply(concat_botpair, axis=1)
df_all['botpair_sorted'] = df_all.apply(sorted_botpair, axis=1)
Explanation: Functions for calculating botpair and botpair_sorted
End of explanation
gb_lpb = df_all.groupby(["language", "rev_page", "botpair"])
gb_lpb_s = df_all.groupby(["language", "rev_page", "botpair_sorted"])
df_lpb = pd.DataFrame(gb_lpb['rev_id'].count()).reset_index().rename(columns={"rev_id":"reverts_per_page_botpair"})
df_lpb[0:5]
df_lpb_s = pd.DataFrame(gb_lpb_s['rev_id'].count()).reset_index().rename(columns={"rev_id":"reverts_per_page_botpair_sorted"})
df_lpb_s[0:5]
df_all = pd.merge(df_all, df_lpb, how='left',
         left_on=["language", "rev_page", "botpair"],
         right_on=["language", "rev_page", "botpair"])
df_all = pd.merge(df_all, df_lpb_s, how='left',
         left_on=["language", "rev_page", "botpair_sorted"],
         right_on=["language", "rev_page", "botpair_sorted"])
Explanation: Reverts per page per botpair
This analysis is also replicated in R in reverts_per_page_R.ipynb
End of explanation
len(df_all.query("time_to_revert_days < 0"))
len(df_all.query("time_to_revert_days > 0"))
df_all.query("time_to_revert_days < 0").groupby("language")['rev_id'].count()
df_all.query("time_to_revert_days > 0").groupby("language")['rev_id'].count()
Explanation: Check time to revert for negatives
End of explanation
len(df_all)
df_all.sample(2).transpose()
Explanation: Final data format
End of explanation
!rm ../../datasets/parsed_dataframes/df_all_2016.p* 
df_all.to_pickle("../../datasets/parsed_dataframes/df_all_2016.pickle")
!xz -k -e -9 ../../datasets/parsed_dataframes/df_all_2016.pickle
df_all.to_csv("../../datasets/parsed_dataframes/df_all_2016.tsv", sep="\t")
!xz -k -e -9 ../../datasets/parsed_dataframes/df_all_2016.tsv
end = datetime.datetime.now()
time_to_run = end - start
minutes = int(time_to_run.seconds/60)
seconds = time_to_run.seconds % 60
print("Total runtime: ", minutes, "minutes, ", seconds, "seconds")
Explanation: Output to file
End of explanation
<END_TASK>