<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Getting Started with CrowdTruth metrics
This tutorial will explain how to use CrowdTruth metrics to process data that was collected with crowdsourcing. For more information about the metrics and how they work, read this paper.
Installing the library
First, you will need to install the CrowdTruth Python package.
To get the stable version from PyPi, install the package using pip
Step1:  To use the development version, download the source code from the GitHub repository and install it frome the console using
Step2:  Example crowdsourcing task
Step3:  Declaring a pre-processing configuration
The pre-processing configuration defines how to interpret the raw crowdsourcing input. To do this, we need to define a configuration class. First, we import the default CrowdTruth configuration class
Step4:  Our test class inherits the default configuration DefaultConfig, while also declaring some additional attributes that are specific to the Medical Relation Extraction task
Step5:  Pre-processing the input data
After declaring the configuration of our input file, we are ready to pre-process the crowd data
Step6:  Computing the CrowdTruth metrics
The pre-processed data can then be used to calculate the CrowdTruth metrics
Step7:  results is a dict object that contains the quality metrics for sentences, relations and crowd workers.
The sentence metrics are stored in results["units"]
Step8:  The uqs column in results["units"] contains the sentence quality scores, capturing the overall workers agreement over each sentence. Here we plot its histogram
Step9:  The unit_annotation_score column in results["units"] contains the sentence-relation scores. For each sentence, we store a dict of relations
Step10:  The worker metrics are stored in results["workers"]
Step11:  The wqs columns in results["workers"] contains the worker quality scores, capturing the overall agreement between one worker and all the other workers.
Step12:  The relation metrics are stored in results["annotations"]. The aqs column contains the relation quality scores, capturing the overall worker agreement over one relation.
<ASSISTANT_TASK:>
Python Code:
!pip install crowdtruth
Explanation: Getting Started with CrowdTruth metrics
This tutorial will explain how to use CrowdTruth metrics to process data that was collected with crowdsourcing. For more information about the metrics and how they work, read this paper.
Installing the library
First, you will need to install the CrowdTruth Python package.
To get the stable version from PyPi, install the package using pip:
End of explanation
import crowdtruth
Explanation: To use the development version, download the source code from the GitHub repository and install it frome the console using:
python setup.py install
Now you can load the CrowdTruth library:
End of explanation
import pandas as pd
test_data = pd.read_csv("relex_example.csv")
test_data.head()
Explanation: Example crowdsourcing task: Medical Relation Extraction
In this tutorial, we will work with data from a crowdsourcing task on Medical Relation Extraction. This is a multiple choice task that was executed on FigureEight. The workers were asked to read a medical sentence with 2 highlighted terms, then pick from a multiple choice list what are the relations expressed between the 2 terms in the sentence. Below you can see the task template:
<img src="https://raw.githubusercontent.com/CrowdTruth/Medical-Relation-Extraction/master/img/medical_relex.png" alt="Task Template" style="width: 450px; float: left; border:1px solid #d3d3d3; border-radius: 10px; padding: 1px;" />
A sample dataset for this task is available in this file. containing raw output from the crowd on FigureEight. Download the file and place it in the same folder as this notebook, then check your data:
End of explanation
from crowdtruth.configuration import DefaultConfig
Explanation: Declaring a pre-processing configuration
The pre-processing configuration defines how to interpret the raw crowdsourcing input. To do this, we need to define a configuration class. First, we import the default CrowdTruth configuration class:
End of explanation
class TestConfig(DefaultConfig):
    inputColumns = ["term1", "b1", "e1", "term2", "b2", "e2", "sentence"]
    outputColumns = ["relations"]
    
    annotation_separator = " "
    
    # processing of a closed task
    open_ended_task = False
    annotation_vector = [
        "causes", "manifestation", "treats", "prevents", "symptom", "diagnose_by_test_or_drug",
        "location", "side_effect", "contraindicates", "associated_with", "is_a", "part_of",
        "other", "none"]
    
    def processJudgments(self, judgments):
        # pre-process output to match the values in annotation_vector
        for col in self.outputColumns:
            # remove square brackets from annotations
            judgments[col] = judgments[col].apply(lambda x: str(x).replace('[',''))
            judgments[col] = judgments[col].apply(lambda x: str(x).replace(']',''))
            # transform to lowercase
            judgments[col] = judgments[col].apply(lambda x: str(x).lower())
        return judgments
Explanation: Our test class inherits the default configuration DefaultConfig, while also declaring some additional attributes that are specific to the Medical Relation Extraction task:
inputColumns: list of input columns from the .csv file with the input data
outputColumns: list of output columns from the .csv file with the answers from the workers
annotation_separator: string that separates between the crowd annotations in outputColumns
open_ended_task: boolean variable defining whether the task is open-ended (i.e. the possible crowd annotations are not known beforehand, like in the case of free text input); in the task that we are processing, workers pick the answers from a pre-defined list, therefore the task is not open ended, and this variable is set to False
annotation_vector: list of possible crowd answers, mandatory to declare when open_ended_task is False; for our task, this is the list of medical relations
processJudgments: method that defines processing of the raw crowd data; for this task, we process the crowd answers to correspond to the values in annotation_vector
The complete configuration class is declared below:
End of explanation
data, config = crowdtruth.load(
    file = "relex_example.csv",
    config = TestConfig()
)
data['judgments'].head()
Explanation: Pre-processing the input data
After declaring the configuration of our input file, we are ready to pre-process the crowd data:
End of explanation
results = crowdtruth.run(data, config)
Explanation: Computing the CrowdTruth metrics
The pre-processed data can then be used to calculate the CrowdTruth metrics:
End of explanation
results["units"].head()
Explanation: results is a dict object that contains the quality metrics for sentences, relations and crowd workers.
The sentence metrics are stored in results["units"]:
End of explanation
import matplotlib.pyplot as plt
%matplotlib inline
plt.hist(results["units"]["uqs"])
plt.xlabel("Sentence Quality Score")
plt.ylabel("Sentences")
Explanation: The uqs column in results["units"] contains the sentence quality scores, capturing the overall workers agreement over each sentence. Here we plot its histogram:
End of explanation
results["units"]["unit_annotation_score"].head()
Explanation: The unit_annotation_score column in results["units"] contains the sentence-relation scores. For each sentence, we store a dict of relations
End of explanation
results["workers"].head()
Explanation: The worker metrics are stored in results["workers"]:
End of explanation
plt.hist(results["workers"]["wqs"])
plt.xlabel("Worker Quality Score")
plt.ylabel("Workers")
Explanation: The wqs columns in results["workers"] contains the worker quality scores, capturing the overall agreement between one worker and all the other workers.
End of explanation
results["annotations"]
Explanation: The relation metrics are stored in results["annotations"]. The aqs column contains the relation quality scores, capturing the overall worker agreement over one relation.
End of explanation
<END_TASK>