Augmenting Chest X-ray Datasets with Non-Expert Annotations
Cathrine Damgaard Trine Naja Eriksen Dovile Juodelyte Veronika Cheplygina Amelia Jiménez-Sánchez
The advancement of machine learning algorithms in medical image analysis
requires the expansion of training datasets. A popular and cost-effective
approach is automated annotation extraction from free-text medical reports,
primarily due to the high costs associated with expert clinicians annotating
chest X-ray images. However, it has been shown that the resulting datasets are
susceptible to biases and shortcuts. Another strategy to increase the size of a
dataset is crowdsourcing, a widely adopted practice in general computer vision
with some success in medical image analysis. In a similar vein to
crowdsourcing, we enhance two publicly available chest X-ray datasets by
incorporating non-expert annotations. However, instead of using diagnostic
labels, we annotate shortcuts in the form of tubes. We collect 3.5k chest drain
annotations for CXR14, and 1k annotations for 4 different tube types in
PadChest. We train a chest drain detector with the non-expert annotations that
generalizes well to expert labels. Moreover, we compare our annotations to
those provided by experts and show "moderate" to "almost perfect" agreement.
Finally, we present a pathology agreement study to raise awareness about ground
truth annotations. We make our annotations and code available.