<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Model Checking
Step1:  Q
Step2:  Model does not assign 0 scores, like the annotators.
Model score distribution is skewed left for x > 0.2.
Q
Step3:  Q
Step4:  Q
Step5:  Q
<ASSISTANT_TASK:>
Python Code:
%load_ext autoreload
%autoreload 2
%matplotlib inline
import warnings
warnings.filterwarnings('ignore')
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import pandas as pd
from load_utils import *
d = load_diffs(keep_diff = True)
df_events, df_blocked_user_text = load_block_events_and_users()
Explanation: Model Checking
End of explanation
import matplotlib.pyplot as plt
import numpy as np
fig = plt.figure(figsize=(12, 6))
xa = np.sort(d['annotated']['recipient_score'])
ya = 1. * np.arange(len(xa)) / (len(xa) - 1)
xm = np.sort(d['annotated']['pred_recipient_score'])
ym = 1. * np.arange(len(xm)) / (len(xm) - 1)
ax1 = fig.add_subplot(121)
ax1.plot(xa, ya, label = 'annotators')
ax1.plot(xm, ym, label = 'model')
ax1.set_xlabel('$P(X<x)$')
ax1.set_ylabel('$p$')
ax1.legend()
ax2 = fig.add_subplot(122)
ax2.plot(xa, ya, label = 'annotators')
ax2.plot(xm, ym, label = 'model')
ax2.set_xlabel('$P(X<x)$')
ax2.set_ylabel('$p$')
ax2.set_ylim((0.90, 1))
ax2.legend()
Explanation: Q: How does the distribution over attack probabilities compare across annotators vs models?
End of explanation
import matplotlib.pyplot as plt
import numpy as np
fig = plt.figure(figsize=(12, 6))
xa = np.sort(d['annotated']['recipient_score'])
ya = 100 * (1 - (1. * np.arange(len(xa)) / (len(xa) - 1)))
xm = np.sort(d['annotated']['pred_recipient_score'])
ym = 100 * (1 - (1. * np.arange(len(xm)) / (len(xm) - 1)))
ax1 = fig.add_subplot(121)
ax1.plot(xa, ya, label = 'annotators')
ax1.plot(xm, ym, label = 'model')
ax1.set_xlabel('$t$')
ax1.set_ylabel('Percent of comments that are attacks')
ax1.legend()
ax2 = fig.add_subplot(122)
ax2.plot(xa, ya, label = 'annotators')
ax2.plot(xm, ym, label = 'model')
ax2.set_xlabel('$t$')
ax2.set_ylabel('Percent of comments above threshold t')
ax2.set_ylim((0.0, 3))
ax2.set_xlim((0.2, 1.0))
ax2.legend()
plt.savefig('../../paper/figs/model_vs_annotator_percent_attack_distributions.png')
Explanation: Model does not assign 0 scores, like the annotators.
Model score distribution is skewed left for x > 0.2.
Q: How do differences in distribution over attack probabilities between annotators and models impact estomates of prevalence of personal attacks?
End of explanation
def check_range(d, col, min, max, n = 10, ):
    df = d.query('%s >= %f and %s <= %f' % (col, min, col, max))
    for i, r in df.sample(n).iterrows():
        print(r[col])
        print(r.clean_diff)
        print('\n')  
check_range(d['sample'], 'pred_aggression_score',0.6, 0.7)
Explanation: Q: What do comments in different score intervals look like?
End of explanation
# annotations
sns.distplot(d['annotated']['aggression_score'].dropna(), hist=False, label = 'annotator scores')
# model on annotated data
sns.distplot(d['annotated']['pred_aggression_score'].dropna(), hist=False, label = 'model on annoted data')
#model no admin sample
sns.distplot(d['sample']['pred_aggression_score'].dropna(), hist=False, label = 'model sample')
plt.xlim(-1, 1)
plt.legend()
Explanation: Q: How does the distribution over aggression scores compare across annotators vs models and different data sets
Plot kernel density estimate of aggression score distribution. This is mainly a sanity check that the aggression model is working new data
End of explanation
sns.distplot(d['annotated']['recipient_score'].dropna(), kde =False, norm_hist = True)
sns.distplot(d['annotated']['pred_recipient_score'].dropna(), kde =False, norm_hist = True)
sns.distplot(d['sample']['pred_recipient_score'].dropna(), kde =False, norm_hist = True)
Explanation: Q: How does the distribution over attack probabilities compare across annotators vs models and different data sets
This is mainly a sanity check that the attack model is working new data
End of explanation
<END_TASK>