<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Annotated PMI Keywords with Categories
In this notebook we evaluate the differences in PMI keywords for each gender. We load the data from the previous notebooks; recall that we load PMI data with annotated categories. We test differences in those categories using a chi-square test.
Code by Eduardo Graells-Garrido.
Annotation by Claudia Wagner and Eduardo Graells-Garrido
Step1:  Load Data
Some bigrams include apostrophes at the end. Here we recover their original form.
Step2:  Here we load the DataFrame from the previous notebook. Note that there is an additional column cat.
Step3:  Test Proportions and Effect Size
We test proportions of categories for each gender and then we estimate Conhen's w as effect size.
Step4:  Word clouds
To visualize and explore the distributions of words per category we use word clouds. In particular we use the matta library.
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import re
import numpy as np
import dbpedia_config
from scipy.stats import chisquare
target_folder = dbpedia_config.TARGET_FOLDER
Explanation: Annotated PMI Keywords with Categories
In this notebook we evaluate the differences in PMI keywords for each gender. We load the data from the previous notebooks; recall that we load PMI data with annotated categories. We test differences in those categories using a chi-square test.
Code by Eduardo Graells-Garrido.
Annotation by Claudia Wagner and Eduardo Graells-Garrido
End of explanation
apost = re.compile('_s$')
Explanation: Load Data
Some bigrams include apostrophes at the end. Here we recover their original form.
End of explanation
female_pmi = pd.read_csv('{0}/top-200-pmi-female.csv'.format(target_folder), encoding='utf-8')
female_pmi.word = female_pmi.word.map(lambda x: apost.sub('\'s', x))
female_pmi.head()
female_pmi.cat.value_counts() / female_pmi.shape[0] * 100.0
male_pmi = pd.read_csv('{0}/top-200-pmi-male.csv'.format(target_folder), encoding='utf-8')
male_pmi.word = male_pmi.word.map(lambda x: apost.sub('\'s', x))
male_pmi.head()
male_pmi.cat.value_counts() / male_pmi.shape[0] * 100.0
Explanation: Here we load the DataFrame from the previous notebook. Note that there is an additional column cat.
End of explanation
m_proportions = []
f_proportions = []
m_count = male_pmi.cat.value_counts() / male_pmi.shape[0] * 100.0
f_count = female_pmi.cat.value_counts() / female_pmi.shape[0] * 100.0
for c in ('F', 'G', 'O', 'R'):
    m_proportions.append(m_count[c] if c in m_count.index else 0.0)
    f_proportions.append(f_count[c] if c in f_count.index else 0.0)
    
m_proportions, f_proportions
chisquare(m_proportions, f_proportions)
p0 = np.array(m_proportions)
p1 = np.array(f_proportions)
np.sqrt(np.sum(np.power(p1 - p0, 2) / p1))
Explanation: Test Proportions and Effect Size
We test proportions of categories for each gender and then we estimate Conhen's w as effect size.
End of explanation
import matta
matta.init_javascript(path='https://rawgit.com/carnby/matta/master/matta/libs')
matta.wordcloud(dataframe=female_pmi.loc[:, ('word', 'pmi_female', 'cat')], text='word',
                typeface='Lato', font_weight='bold',
               font_size={'value': 'pmi_female'},
               font_color={'palette': 'Set2', 'n_colors': 4, 'value': 'cat', 'scale': 'ordinal'})
matta.wordcloud(dataframe=male_pmi.loc[:, ('word', 'pmi_male', 'cat')], text='word',
                typeface='Lato', font_weight='bold',
               font_size={'value': 'pmi_male'},
               font_color={'palette': 'Set2', 'n_colors': 4, 'value': 'cat', 'scale': 'ordinal'})
Explanation: Word clouds
To visualize and explore the distributions of words per category we use word clouds. In particular we use the matta library.
End of explanation
<END_TASK>