Label Selection Approach to Learning from Crowds
Kosuke Yoshimura Hisashi Kashima
Supervised learning, especially supervised deep learning, requires large
amounts of labeled data. One approach to collect large amounts of labeled data
is by using a crowdsourcing platform where numerous workers perform the
annotation tasks. However, the annotation results often contain label noise, as
the annotation skills vary depending on the crowd workers and their ability to
complete the task correctly. Learning from Crowds is a framework which directly
trains the models using noisy labeled data from crowd workers. In this study,
we propose a novel Learning from Crowds model, inspired by SelectiveNet
proposed for the selective prediction problem. The proposed method called Label
Selection Layer trains a prediction model by automatically determining whether
to use a worker's label for training using a selector network. A major
advantage of the proposed method is that it can be applied to almost all
variants of supervised learning problems by simply adding a selector network
and changing the objective function for existing models, without explicitly
assuming a model of the noise in crowd annotations. The experimental results
show that the performance of the proposed method is almost equivalent to or
better than the Crowd Layer, which is one of the state-of-the-art methods for
Deep Learning from Crowds, except for the regression problem case.