An Empirical Study of Uncertainty in Polygon Annotation and the Impact
  of Quality Assurance
Eric Zimmermann Justin Szeto Frederic Ratle
Polygons are a common annotation format used for quickly annotating objects
in instance segmentation tasks. However, many real-world annotation projects
request near pixel-perfect labels. While strict pixel guidelines may appear to
be the solution to a successful project, practitioners often fail to assess the
feasibility of the work requested, and overlook common factors that may
challenge the notion of quality. This paper aims to examine and quantify the
inherent uncertainty for polygon annotations and the role that quality
assurance plays in minimizing its effect. To this end, we conduct an analysis
on multi-rater polygon annotations for several objects from the MS-COCO
dataset. The results demonstrate that the reliability of a polygon annotation
is dependent on a reviewing procedure, as well as the scene and shape
complexity.