Inter-Annotator Agreement in the Wild: Uncovering Its Emerging Roles and
  Considerations in Real-World Scenarios
NamHyeok Kim Chanjun Park
Inter-Annotator Agreement (IAA) is commonly used as a measure of label
consistency in natural language processing tasks. However, in real-world
scenarios, IAA has various roles and implications beyond its traditional usage.
In this paper, we not only consider IAA as a measure of consistency but also as
a versatile tool that can be effectively utilized in practical applications.
Moreover, we discuss various considerations and potential concerns when
applying IAA and suggest strategies for effectively navigating these
challenges.