Calculating Inter Annotator Agreement with brat annotated files
<p>With three annotators we have been using brat (<a href="http://brat.nlplab.org/" rel="nofollow noreferrer">http://brat.nlplab.org/</a>) to annotate a sample of texts for three categories: PERS, ORG, GPE. I want to calculate Inter Annotator Agreement for that sample. However, I do not seem to find a simple way to do it. I have tried this Python package: <a href="https://github.com/savkov/BratUtils" rel="nofollow noreferrer">https://github.com/savkov/BratUtils</a> , but it seems to fail. </p>

<p>For each annotated sample I have three .ann files for which I want to calculate Inter Annotator Agreement. The data in the files looks like this:</p>

<pre><code>T1  Person 173 183  Alexanders
T2  Person 444 450  Hannah
T3  Person 754 766  Corneliszoon
T4  GPE 787 796 Antwerpen
T5  Person 824 833  Alexander
</code></pre>

<p>Is there a simple way to calculate Inter Annotator Agreement (using Python or a web-based tool)? </p>