Your Model Is Not Predicting Depression Well And That Is Why: A Case
  Study of PRIMATE Dataset
Kirill Milintsevich Kairit Sirts Gaël Dias
This paper addresses the quality of annotations in mental health datasets
used for NLP-based depression level estimation from social media texts. While
previous research relies on social media-based datasets annotated with binary
categories, i.e. depressed or non-depressed, recent datasets such as D2S and
PRIMATE aim for nuanced annotations using PHQ-9 symptoms. However, most of
these datasets rely on crowd workers without the domain knowledge for
annotation. Focusing on the PRIMATE dataset, our study reveals concerns
regarding annotation validity, particularly for the lack of interest or
pleasure symptom. Through reannotation by a mental health professional, we
introduce finer labels and textual spans as evidence, identifying a notable
number of false positives. Our refined annotations, to be released under a Data
Use Agreement, offer a higher-quality test set for anhedonia detection. This
study underscores the necessity of addressing annotation quality issues in
mental health datasets, advocating for improved methodologies to enhance NLP
model reliability in mental health assessments.