Skin Deep: Investigating Subjectivity in Skin Tone Annotations for
  Computer Vision Benchmark Datasets
Teanna Barrett Quan Ze Chen Amy X. Zhang
To investigate the well-observed racial disparities in computer vision
systems that analyze images of humans, researchers have turned to skin tone as
more objective annotation than race metadata for fairness performance
evaluations. However, the current state of skin tone annotation procedures is
highly varied. For instance, researchers use a range of untested scales and
skin tone categories, have unclear annotation procedures, and provide
inadequate analyses of uncertainty. In addition, little attention is paid to
the positionality of the humans involved in the annotation process--both
designers and annotators alike--and the historical and sociological context of
skin tone in the United States. Our work is the first to investigate the skin
tone annotation process as a sociotechnical project. We surveyed recent skin
tone annotation procedures and conducted annotation experiments to examine how
subjective understandings of skin tone are embedded in skin tone annotation
procedures. Our systematic literature review revealed the uninterrogated
association between skin tone and race and the limited effort to analyze
annotator uncertainty in current procedures for skin tone annotation in
computer vision evaluation. Our experiments demonstrated that design decisions
in the annotation procedure such as the order in which the skin tone scale is
presented or additional context in the image (i.e., presence of a face)
significantly affected the resulting inter-annotator agreement and individual
uncertainty of skin tone annotations. We call for greater reflexivity in the
design, analysis, and documentation of procedures for evaluation using skin
tone.