<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Question 1
Step1:  Question 2
Step2:  Question 3
Step3:  Question 4
Step4:  Question 5
Step5:  Question 6
Step6:  To quantitatively measure the linear correlation between number of ratings and average correctness, linear regression is used to draw insights.
From the model summary, it is still difficult to establish reliable linear correlation between the two variables, since the coefficient of number of ratings is not significantly different from zero.
statsmodels and patsy modules are imported for linear regression
Step7:  Question 7
Step8:  Question 8
Step9:  Question 9
<ASSISTANT_TASK:>
Python Code:
gold = pd.read_table("gold.txt", names=["url", "category"]).dropna()
labels = pd.read_table("labels.txt", names=["turk", "url", "category"]).dropna()
Explanation: Question 1: Read in data
Read in the data from "gold.txt" and "labels.txt".
Since there are no headers in the files, names parameter should be set explicitly.
Duplicate records in both dataframes are kept, for repeated test on the same url provides enables more precise information about the turks' discernibility
End of explanation
url_list = gold["url"].unique()
labels_on_gold = labels[labels["url"].map(lambda s: s in url_list)]
labels_unknown = labels[labels["url"].map(lambda s: s not in url_list)]
Explanation: Question 2: Split into two DataFrames
To determine if a url in labels is in gold, make a list of unique url in gold, and map the lambda expression on the url series in labels.
End of explanation
rater_merged = pd.merge(
                    labels_on_gold,
                    gold,
                    left_on="url",
                    right_on="url",
                    suffixes=["_1", "_2"]
                )
rater_merged["correct"] = rater_merged["category_1"] == rater_merged["category_2"]
rater_merged = rater_merged[["turk", "correct"]]
correct_counts = rater_merged.groupby("turk")["correct"].sum()
total_counts = rater_merged["turk"].value_counts()
avg_correctness = correct_counts/total_counts
rater_goodness = pd.DataFrame({"number_of_ratings": total_counts, "average_correctness": avg_correctness})
rater_goodness[:10]
Explanation: Question 3: Compute accuracies of turks
Since the computation is all on "gold" set url, "labels_on_gold" dataframe is used instead of "labels"
Merge "labels_on_gold" with "gold" on url.
Create a new column correct in the new dataframe, and assign True where the "turk" rating is the same with the true rating.
Optional: drop the rating columns to reduce the size of the dataframe
groupby on turk, and sum up the True records on correct for each turk, the returned value is a series
value_counts on turk, a series of total rating numbers is returned.
Divide the previous two series to get the rating accuracy of each turk
Create a new dataframe "rater_goodness" with the total rating number series and rating accuracy series, index by default set as turk
End of explanation
rater_goodness["odds"] = rater_goodness["average_correctness"].map(lambda x: x/(1.001-x))
rater_goodness[:20]
Explanation: Question 4: Odds ratios
Use "map" function on average_correctness to get $\frac{average\ correctness}{1 - average\ correctness}$
By definition, when average_correctness = 1, the ratio should be assigned float("inf")
End of explanation
rater_goodness[rater_goodness["number_of_ratings"]>=20].sort_values(by="average_correctness", ascending=False)[:10]
Explanation: Question 5: Most accurate turks
Use rater_goodness["number of ratings"]&gt;=20 to select turks who rated at least 20 times.
Sort the list by average_correctness in descending order.
.index.values is optional to return only turks, but for aesthetic reasons it is not applied.
End of explanation
plot(rater_goodness['number_of_ratings'],
     rater_goodness['average_correctness'],
     marker='o',
     color='blue',
     linestyle='None')
xlabel('number of ratings')
ylabel('average correctness')
Explanation: Question 6: Rating counts versus accuracy
Plotting average_correctness against number of ratings makes it easier to have an general idea between the two variables. However, from the plot, it is difficult to identify a clear pattern.
End of explanation
import statsmodels.api as sm
from patsy import dmatrices
y, X = dmatrices('average_correctness ~ number_of_ratings', data=rater_goodness, return_type='dataframe')
model = sm.OLS(y, X)
result = model.fit()
print result.summary()
Explanation: To quantitatively measure the linear correlation between number of ratings and average correctness, linear regression is used to draw insights.
From the model summary, it is still difficult to establish reliable linear correlation between the two variables, since the coefficient of number of ratings is not significantly different from zero.
statsmodels and patsy modules are imported for linear regression
End of explanation
top_25_cutpoint = labels_on_gold["turk"].value_counts().quantile(q=.75)
turk_list = labels_on_gold["turk"].value_counts()
mask_1 = labels_unknown["turk"].map(lambda s: turk_list[s]>=top_25_cutpoint if s in turk_list else False)
labels_bytop25 = labels_unknown[mask_1]
rater_goodness["turk"] = rater_goodness.index
odds_top25 = rater_goodness[rater_goodness["turk"].map(lambda s: turk_list[s]>=top_25_cutpoint if s in turk_list else False)]
overall_odds = pd.merge(labels_bytop25,
                       odds_top25,
                       left_on="turk",
                       right_on="turk",
                       how="left").dropna()
overall_odds.groupby(["url", "category"])[["odds"]].prod()[:10]
Explanation: Question 7: Overall predicted odds
Define the cutpoint of top 25% turks in term of number of ratings using quantile(q=.75).
Make a list of "turk: number of ratings"
Make a mask to select records rated by top 25% turks using map function.
Select from the total "labels" data set the records rated by top 25% turks.
Merge this dataframe with "labels_unknown" dataframe on url and category, duplicates dropped.
Next merge the resulting dataframe with "rater_goodness" dataframe.
First create a new turk column in "rater_goodness" dataframe from the index
Only select the records rated by top 25% turks from "rater_goodness" dataframe
Merge the two dataframe on turk
Drop duplicates and missing values
groupby the resulting dataframe on url and category.
Apply prod() on odds to calculate overall odds by url and category.
here odds is the "overall odds" as defined in the assignment description
End of explanation
overall_odds_df = overall_odds.groupby(["url", "category"])[["odds"]].prod().unstack("category").T.fillna(0)
url_rating = pd.DataFrame(overall_odds_df.idxmax())
url_rating["top category"] = url_rating[0].map(lambda s: s[1])
url_rating = url_rating.set_index(url_rating.index.values)
url_rating["top odds"] = overall_odds_df.max()
url_rating = url_rating[["top category", "top odds"]]
url_rating[:10]
Explanation: Question 8: Predicted categories
Create a dataframe from the groupby object in the last question, containing url, category and overall odds.
Apply unstack to breakdown category from index to columns.
Transpose the dataframe and get idxmax() on all columns, i.e. url, returned value is a series with url as index and np.array ("odds", category) as values.
Create a dataframe using the returned series, and convert the np.array into a string column "top category" by selecting the second element.
Create a new "top odds" column for the dataframe by max() on the transposed dataframe in step 2.
End of explanation
top_75_cutpoint = labels_on_gold["turk"].value_counts().quantile(q=.25)
mask_2 = labels_unknown["turk"].map(lambda s: turk_list[s]>=top_75_cutpoint if s in turk_list else False)
labels_bytop75 = labels_unknown[mask_2]
odds_top75 = rater_goodness[rater_goodness["turk"].map(lambda s: turk_list[s]>=top_75_cutpoint if s in turk_list else False)]
overall_odds_75 = pd.merge(labels_bytop75,
                       odds_top75,
                       left_on="turk",
                       right_on="turk",
                       how="left").dropna()
overall_odds_df_75 = overall_odds_75.groupby(["url", "category"])[["odds"]].prod().unstack("category").T.fillna(0)
url_rating_75 = pd.DataFrame(overall_odds_df_75.idxmax())
url_rating_75["result_75"] = url_rating_75[0].map(lambda s: s[1])
url_rating_75 = pd.DataFrame(url_rating_75["result_75"])
url_rating_75 = url_rating_75.set_index(url_rating_75.index.values)
url_rating_25 = pd.DataFrame({"result_25": url_rating["top category"]})
url_rating_merged = pd.merge(url_rating_25,
                            url_rating_75,
                            left_index=True,
                            right_index=True,
                            ).dropna()
url_rating_crosstab = pd.crosstab(index=url_rating_merged["result_25"],
                                 columns=url_rating_merged["result_75"]
                                 )
url_rating_crosstab
Explanation: Question 9: Predicted categories using more turks
Repeat Question\ 7 and Question\ 8 to create a dataframe where url are rated by top 75% turks.
    > Here only the "top category" column is kept and named result_75
Take out top category column from the dataframe from Question 8 and rename it result_25, and make it a dataframe.
Merge the two dataframes on index.
Create a crosstab with the two columns as index and columns respectively.
From the crosstab it can be seen that, the most errors are where the top 25% turks rated "G" but the top 75% turks rated "P" (836 occurences), "G" versus "R" (285 occurences), and "P" versus "G" (229 occurences).
End of explanation
<END_TASK>