<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Introduction
This notebook is an analysis of the Crowdflower labels of 1000 random revisions of Wikipedia talk pages. This dataset has been cleaned and filtered to remove common administrator messages. These datasets are annotated via crowdflower to measure friendliness, aggressiveness and whether the comment constitutes a personal attack. 
On Crowdflower, each revision is rated 7 times. The raters are given two questions
Step1:  Plot histogram of average ratings by revision
For each revision, we take the average of all the ratings by level of friendliness/aggressiveness and for each of the answers to Question 3. The histograms of these averages is displayed below.
Step2:  Selected harassing and aggressive revisions by quartile
We look at a sample of revisions whose average aggressive score falls into various quantiles. This allows us to subjectively evaluate the quality of the questions that we are asking on Crowdflower.
Step3:  Inter-Annotator Agreement
Below, we compute the Krippendorf's Alpha, which is a measure of the inter-annotator agreement of our Crowdflower responses. We achieve an Alpha value of 0.668 on our dataset, which is a relatively good level of inter-annotator agreement for this type of subjective inquiry.
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
from __future__ import division
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
pd.set_option('display.max_colwidth', 1000)
# Download data from google drive (Respect Eng / Wiki Collab): wikipdia data/v0_annotated
dat = pd.read_csv('../data/experiment-0.csv')
# Remove test questions
dat = dat[dat['_golden'] == False]
dat['num_aggressive_na'] = np.isnan(dat['how_aggressive_or_friendly'])
dat['num_attack_na'] = np.isnan(dat['is_harassment_or_attack'])
def create_column_of_counts_from_nums(df, col):
    return df.apply(lambda x: int(col) == x)
aggressive_columns = ['-2', '-1', '0', '1', '2']
for col in aggressive_columns:
    dat[col] = create_column_of_counts_from_nums(dat['how_aggressive_or_friendly'], col)
dat['not_attack_1'] = 1 - dat['is_harassment_or_attack']
dat['not_attack_0'] = dat['is_harassment_or_attack']
# Group the data
agg_dict = dict.fromkeys(aggressive_columns, 'sum')
agg_dict.update({'clean_diff': 'first', 'num_aggressive_na': 'sum', 'num_attack_na': 'sum',
                 '_id':'count', 'not_attack_0':'sum', 'not_attack_1': 'sum'})
grouped_dat = dat.groupby(['rev_id'], as_index=False).agg(agg_dict)
# Get rid of data which the majority thinks is not in English or not readable
grouped_dat = grouped_dat[(grouped_dat['num_aggressive_na'] <= grouped_dat['_id']/2) & (grouped_dat['num_attack_na'] <= grouped_dat['_id']/2)]
def compute_aggression_score(x, cols):
    n = 0
    s = 0
    for col in cols:
        s = s + x[col]*int(col)
        n = n + x[col]
    return s/n
grouped_dat['aggression_score'] = grouped_dat[aggressive_columns].dot([-2,-1,0,1,2])/grouped_dat['_id']
grouped_dat['attack_score'] = grouped_dat['not_attack_0']/grouped_dat['_id']
Explanation: Introduction
This notebook is an analysis of the Crowdflower labels of 1000 random revisions of Wikipedia talk pages. This dataset has been cleaned and filtered to remove common administrator messages. These datasets are annotated via crowdflower to measure friendliness, aggressiveness and whether the comment constitutes a personal attack. 
On Crowdflower, each revision is rated 7 times. The raters are given two questions:
How aggressive or friendly is the tone of this comment?
Ranges from ‘Very Aggressive’ (2) to ‘Very Friendly’ (-2)
Includes an ‘N/A’ value for comments that aren’t in English or human-readable.
Is this an example of harassment or a personal attack?
Values are ‘Yes’ (1), ‘No’ (0), and ‘N/A’ (‘N/A’)
Below, we plot histograms of the units by average rating of each of the questions, examine quantiles of answers, and compute inter-annotator agreement.
Loading packages and data
End of explanation
def hist_comments(df, bins, plot_by, title):
    plt.figure()
    sliced_array = df[[plot_by]]
    weights = np.ones_like(sliced_array)/len(sliced_array)
    sliced_array.plot.hist(bins = bins, legend = False, title = title, weights=weights)
    plt.ylabel('Proportion')
    plt.xlabel('Average Score')
bins = np.linspace(-3,3,11)
hist_comments(grouped_dat, bins, 'aggression_score', 'Average Aggressiveness Rating')
bins = np.linspace(0,1,9)
for col in ['attack_score']:
    hist_comments(grouped_dat, bins, col, 'Average Attack Rating')
Explanation: Plot histogram of average ratings by revision
For each revision, we take the average of all the ratings by level of friendliness/aggressiveness and for each of the answers to Question 3. The histograms of these averages is displayed below.
End of explanation
def sorted_comments(df, sort_by, quartile, num, is_ascending = True):
    n = df.shape[0]
    start_index = int(quartile*n)
    return df[['clean_diff', 'aggression_score', 'attack_score']].sort_values(
        by=sort_by, ascending = is_ascending)[start_index:start_index + num]
# Least aggressive comments
sorted_comments(grouped_dat, 'aggression_score', 0, 5)
# Median aggressive comments
sorted_comments(grouped_dat, 'aggression_score', 0.5, 5)
# Most aggressive comments
sorted_comments(grouped_dat, 'aggression_score', 0, 5, False)
Explanation: Selected harassing and aggressive revisions by quartile
We look at a sample of revisions whose average aggressive score falls into various quantiles. This allows us to subjectively evaluate the quality of the questions that we are asking on Crowdflower.
End of explanation
def add_row_to_coincidence(o, row, columns):
    m_u = row.sum(1)
    for i in columns:
        for j in columns:
            if i == j:
                o[i][j] = o[i][j] + row[i]*(row[i]-1)/(m_u-1)
            else:
                o[i][j] = o[i][j] + row[i]*row[j]/(m_u-1)
    return o
def make_coincidence_matrix(df, columns):
    df = df[columns]
    n = df.shape[0]
    num_cols = len(columns)
    o = pd.DataFrame(np.zeros((num_cols,num_cols)), index = columns, columns=columns)
    for i in xrange(n):
        o = add_row_to_coincidence(o, df[i:i+1], columns)
    return o
def binary_distance(i,j):
    return i!=j
def interval_distance(i,j):
    return (int(i)-int(j))**2
def e(n, i, j):
    if i == j:
        return n[i]*(n[i]-1)/sum(n)-1
    else:
        return n[i]*n[j]/sum(n)-1
def D_e(o, columns, distance):
    n = o.sum(1)
    output = 0
    for i in columns:
        for j in columns:
            output = output + e(n,i,j)*distance(i,j)
    return output
def D_o(o, columns, distance):
    output = 0
    for i in columns:
        for j in columns:
            output = output + o[i][j]*distance(i,j)
    return output
def Krippendorf_alpha(df, columns, distance = binary_distance, o = None):
    if o is None:
        o = make_coincidence_matrix(df, columns)
    d_o = D_o(o, columns, distance)
    d_e = D_e(o, columns, distance)
    return (1 - d_o/d_e)
print "Krippendorf's Alpha (aggressiveness): "
Krippendorf_alpha(grouped_dat, aggressive_columns, distance = interval_distance)
print "Krippendorf's Alpha (attack): "
Krippendorf_alpha(grouped_dat, ['not_attack_0', 'not_attack_1'])
Explanation: Inter-Annotator Agreement
Below, we compute the Krippendorf's Alpha, which is a measure of the inter-annotator agreement of our Crowdflower responses. We achieve an Alpha value of 0.668 on our dataset, which is a relatively good level of inter-annotator agreement for this type of subjective inquiry.
End of explanation
<END_TASK>