<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Experimenting with estimating the gender of mailing list participants.
Step1:  Import the BigBang modules as needed. These should be in your Python environment if you've installed BigBang correctly.
Step2:  Now let's load the data for analysis.
Step3:  For each of our lists, we'll clean up the names, find the first name if there is one, and guess its gender. Pandas groups the data together for comparison. We keep count of the names we find that are ambiguous, for the next step.
Step4:  Let's quickly visualize the names that couldn't be guessed with our estimator and their distribution.
Step5:  This distribution may vary by the particular list, but it seems to be a power distribution. That is, with a fairly small supplement of manually providing genders for the names/identities on the list, we can very signficantly improve the fraction of messages with an estimated gender.
With a couple minutes of manual work from someone familiar with the group, I've created an updated CSV that contains a manually-entered gender column, in addition to the automated guess. Let's see how much of a difference that makes.
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
Explanation: Experimenting with estimating the gender of mailing list participants.
End of explanation
import bigbang.ingress.mailman as mailman
import bigbang.analysis.graph as graph
import bigbang.analysis.process as process
from bigbang.parse import get_date
from bigbang.archive import Archive
import imp
imp.reload(process)
import pandas as pd
import datetime
import matplotlib.pyplot as plt
import numpy as np
import math
import pytz
import pickle
import os
from bigbang import parse
from gender_detector import gender_detector as gd
Explanation: Import the BigBang modules as needed. These should be in your Python environment if you've installed BigBang correctly.
End of explanation
urls = [
    "http://www.ietf.org/mail-archive/text/ietf-privacy/",
    "http://lists.w3.org/Archives/Public/public-privacy/",
]
mlists = [(url, mailman.open_list_archives(url,"../../archives")) for url in urls]
Explanation: Now let's load the data for analysis.
End of explanation
detector = gd.GenderDetector('us')
gender_ambiguous_names = {}
def guess_gender(name):
    if not name:
        return 'name unknown'
    try:
        if detector.guess(name) == 'unknown':
            if name in gender_ambiguous_names:
                gender_ambiguous_names[name] += 1
            else:
                gender_ambiguous_names[name] = 1
        
        return detector.guess(name)
    except:
        return 'error'
def ml_shortname(url):
    return url.rstrip("/").split("/")[-1]
series = []  
for (url, ml) in mlists:
    activity = Archive.get_activity(Archive(ml)).sum(0)
    activityFrame = pd.DataFrame(activity, columns=['Message Count'])
    
    activityFrame['Name'] = activityFrame.index.map(lambda x: parse.clean_from(x))    
    activityFrame['First Name'] = activityFrame['Name'].map(lambda x: parse.guess_first_name(x))
    activityFrame['Guessed Gender'] = activityFrame['First Name'].map(guess_gender)
    
    activityFrame.to_csv(('senders_guessed_gender-%s.csv' % ml_shortname(url)),encoding='utf-8')
    
    counts = activityFrame.groupby('Guessed Gender')['Message Count'].count()
    counts.name=url
    series.append(counts)
pd.DataFrame(series)
Explanation: For each of our lists, we'll clean up the names, find the first name if there is one, and guess its gender. Pandas groups the data together for comparison. We keep count of the names we find that are ambiguous, for the next step.
End of explanation
ser = pd.Series(gender_ambiguous_names)
ser.sort_values(ascending=False).plot(kind='bar')
Explanation: Let's quickly visualize the names that couldn't be guessed with our estimator and their distribution.
End of explanation
url = "http://lists.w3.org/Archives/Public/public-privacy/"
csv_guessed = ('senders_guessed_gender-%s.csv' % ml_shortname(url))
csv_manual = ('senders_manual_gender-%s.csv' % ml_shortname(url))
guessed = pd.read_csv(csv_guessed)
manual = pd.read_csv(csv_manual)
def combined_gender(row):
    if str(row['Manual Gender']) != 'nan':
        return row['Manual Gender']
    else:
        return row['Guessed Gender']
manual['Combined Gender'] = manual.apply(combined_gender, axis=1)
combined_series = manual.groupby('Combined Gender')['Message Count'].sum()
guessed_series = manual.groupby('Guessed Gender')['Message Count'].sum()
compared_counts = pd.DataFrame({'Manual':combined_series, 'Guessed':guessed_series})
compared_counts.plot(kind='bar')
figure,axes = plt.subplots(ncols=2, figsize=(8,4))
guessed_series.rename("Guessed").plot(kind='pie', ax=axes[0])
combined_series.rename("Manual").plot(kind='pie', ax=axes[1])
axes[0].set_aspect('equal')
axes[1].set_aspect('equal')
Explanation: This distribution may vary by the particular list, but it seems to be a power distribution. That is, with a fairly small supplement of manually providing genders for the names/identities on the list, we can very signficantly improve the fraction of messages with an estimated gender.
With a couple minutes of manual work from someone familiar with the group, I've created an updated CSV that contains a manually-entered gender column, in addition to the automated guess. Let's see how much of a difference that makes.
End of explanation
<END_TASK>