<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Statistical analysis
Step1:  Statistical analysis on Allsides bias rating
Step2:  from the stats page "For normally distributed data, the skewness should be about 0. A skewness value > 0 means that there is more weight in the left tail of the distribution. The function skewtest can be used to determine if the skewness value is close enough to 0, statistically speaking."
Step3:  Donald Trump Image Box images versus Baseline images source bias according to Facebook bias ratings
Step4:  The Kolmogorov-Smirnov analyis shows that the distribution of political representation across image sources is different between the baseline images and those found in the image box.
Statistical analysis on Allsides + Facebook + MondoTimes + my bias ratings
Step5:  Prepare data for chi squared test
Step6:  Remove Unknown / unreliable row
Step7:  Calculate percentages for plotting purposes
Step8:  Test Hillary Clinton Image Box images against Baseline images
Step9:  Test Donald Trump Image Box images against Basline images
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
%matplotlib inline
plt.style.use('ggplot')
plt.rcParams['figure.figsize'] = (15, 3)
plt.rcParams['font.family'] = 'sans-serif'
pd.set_option('display.width', 5000) 
pd.set_option('display.max_columns', 60)
HC_baseline = pd.read_csv('./BASELINE/HC_baseline_full_ratings.csv')
DT_baseline = pd.read_csv('./BASELINE/DT_baseline_full_ratings.csv')
HC_imagebox = pd.read_csv('./IMAGE_BOX/HC_imagebox_full_ratings.csv')
DT_imagebox = pd.read_csv('./IMAGE_BOX/DT_imagebox_full_ratings.csv')
Explanation: Statistical analysis
End of explanation
print("Baseline skew: ", stats.skew(HC_baseline.facebookbias_rating[HC_baseline.facebookbias_rating<3]))
print("Image Box skew: ", stats.skew(HC_imagebox.facebookbias_rating[HC_imagebox.facebookbias_rating<3]))
Explanation: Statistical analysis on Allsides bias rating:
No sources from the images boxes were rated in the Allsides bias rating dataset. Therefore comparisons between bias of baseline sources versus image box sources could not be performed.
Statistical analysis on Facebook Study bias rating:
Hillary Clinton Image Box images versus Baseline images source bias according to Facebook bias ratings:
End of explanation
print("Baseline skew: ", stats.skewtest(HC_baseline.facebookbias_rating[HC_baseline.facebookbias_rating<3]))
print("Image Box skew: ", stats.skewtest(HC_imagebox.facebookbias_rating[HC_imagebox.facebookbias_rating<3]))
stats.ks_2samp(HC_baseline.facebookbias_rating[HC_baseline.facebookbias_rating<3], 
                               HC_imagebox.facebookbias_rating[HC_imagebox.facebookbias_rating<3])
HC_imagebox.facebookbias_rating.plot.hist(alpha=0.5, bins=20, range=(-1,1), color='blue')
HC_baseline.facebookbias_rating.plot.hist(alpha=0.5, bins=20, range=(-1,1), color='green')
Explanation: from the stats page "For normally distributed data, the skewness should be about 0. A skewness value > 0 means that there is more weight in the left tail of the distribution. The function skewtest can be used to determine if the skewness value is close enough to 0, statistically speaking."
End of explanation
print("Baseline skew: ", stats.skew(DT_baseline.facebookbias_rating[DT_baseline.facebookbias_rating<3]))
print("Image Box skew: ", stats.skew(DT_imagebox.facebookbias_rating[DT_imagebox.facebookbias_rating<3]))
stats.ks_2samp(DT_baseline.facebookbias_rating[DT_baseline.facebookbias_rating<3], 
                               DT_imagebox.facebookbias_rating[DT_imagebox.facebookbias_rating<3])
DT_imagebox.facebookbias_rating.plot.hist(alpha=0.5, bins=20, range=(-1,1), color='red')
DT_baseline.facebookbias_rating.plot.hist(alpha=0.5, bins=20, range=(-1,1), color='green')
print("Number of missing ratings for Hillary Clinton Baseline data: ", len(HC_baseline[HC_baseline.facebookbias_rating == 999]))
print("Number of missing ratings for Hillary Clinton Image Box data: ", len(HC_imagebox[HC_imagebox.facebookbias_rating == 999]))
print("Number of missing ratings for Donald Trump Baseline data: ", len(DT_baseline[DT_baseline.facebookbias_rating == 999]))
print("Number of missing ratings for Donald Trump Image Box data: ", len(DT_baseline[DT_imagebox.facebookbias_rating == 999]))
Explanation: Donald Trump Image Box images versus Baseline images source bias according to Facebook bias ratings:
End of explanation
def convert_to_ints(col):
    if col == 'Left':
        return -1
    elif col == 'Center':
        return 0
    elif col == 'Right':
        return 1
    else:
        return np.nan
HC_imagebox['final_rating_ints'] = HC_imagebox.final_rating.apply(convert_to_ints)
DT_imagebox['final_rating_ints'] = DT_imagebox.final_rating.apply(convert_to_ints)
HC_baseline['final_rating_ints'] = HC_baseline.final_rating.apply(convert_to_ints)
DT_baseline['final_rating_ints'] = DT_baseline.final_rating.apply(convert_to_ints)
HC_imagebox.final_rating_ints.value_counts()
DT_imagebox.final_rating_ints.value_counts()
Explanation: The Kolmogorov-Smirnov analyis shows that the distribution of political representation across image sources is different between the baseline images and those found in the image box.
Statistical analysis on Allsides + Facebook + MondoTimes + my bias ratings:
Convert strings to integers:
End of explanation
HC_baseline_counts = HC_baseline.final_rating.value_counts()
HC_imagebox_counts = HC_imagebox.final_rating.value_counts()
DT_baseline_counts = DT_baseline.final_rating.value_counts()
DT_imagebox_counts = DT_imagebox.final_rating.value_counts()
HC_baseline_counts.head()
normalised_bias_ratings = pd.DataFrame({'HC_ImageBox':HC_imagebox_counts,
                                       'HC_Baseline' : HC_baseline_counts,
                                       'DT_ImageBox': DT_imagebox_counts,
                                       'DT_Baseline': DT_baseline_counts} )
normalised_bias_ratings
Explanation: Prepare data for chi squared test
End of explanation
normalised_bias_ratings = normalised_bias_ratings[:3]
Explanation: Remove Unknown / unreliable row
End of explanation
normalised_bias_ratings.loc[:,'HC_Baseline_pcnt'] = normalised_bias_ratings.HC_Baseline/normalised_bias_ratings.HC_Baseline.sum()*100
normalised_bias_ratings.loc[:,'HC_ImageBox_pcnt'] = normalised_bias_ratings.HC_ImageBox/normalised_bias_ratings.HC_ImageBox.sum()*100
normalised_bias_ratings.loc[:,'DT_Baseline_pcnt'] = normalised_bias_ratings.DT_Baseline/normalised_bias_ratings.DT_Baseline.sum()*100
normalised_bias_ratings.loc[:,'DT_ImageBox_pcnt'] = normalised_bias_ratings.DT_ImageBox/normalised_bias_ratings.DT_ImageBox.sum()*100
normalised_bias_ratings
normalised_bias_ratings.columns
HC_percentages = normalised_bias_ratings[['HC_Baseline_pcnt', 'HC_ImageBox_pcnt']]
DT_percentages = normalised_bias_ratings[['DT_Baseline_pcnt', 'DT_ImageBox_pcnt']]
Explanation: Calculate percentages for plotting purposes
End of explanation
stats.chisquare(f_exp=normalised_bias_ratings.HC_Baseline, 
               f_obs=normalised_bias_ratings.HC_ImageBox)
HC_percentages.plot.bar()
Explanation: Test Hillary Clinton Image Box images against Baseline images:
End of explanation
stats.chisquare(f_exp=normalised_bias_ratings.DT_Baseline, 
               f_obs=normalised_bias_ratings.DT_ImageBox)
DT_percentages.plot.bar()
Explanation: Test Donald Trump Image Box images against Basline images:
End of explanation
<END_TASK>