Towards Label Embedding -- Measuring classification difficulty
Katharina Hechinger Christoph Koller Xiao Xiang Zhu Göran Kauermann
Uncertainty quantification in machine learning is a timely and vast field of
research. In supervised learning, uncertainty can already occur in the very
first stage of the training process, the labelling step. In particular, this is
the case when not every instance can be unambiguously classified. The problem
occurs for classifying instances, where classes may overlap or instances can
not be clearly categorised. In other words, there is inevitable ambiguity in
the annotation step and not necessarily a 'ground truth'. We look exemplary at
the classification of satellite images. Each image is annotated independently
by multiple labellers and classified into local climate zones (LCZs). For each
instance we have multiple votes, leading to a distribution of labels rather
than a single value. The main idea of this work is that we do not assume a
ground truth label but embed the votes into a K-dimensional space, with K as
the number of possible categories. The embedding is derived from the voting
distribution in a Bayesian setup, modelled via a Dirichlet-Multinomial model.
We estimate the model and posteriors using a stochastic Expectation
Maximisation algorithm with Markov Chain Monte Carlo steps. While we focus on
the particular example of LCZ classification, the methods developed in this
paper readily extend to other situations where multiple annotators
independently label texts or images. We also apply our approach to two other
benchmark datasets for image classification to demonstrate this. Besides the
embeddings themselves, we can investigate the resulting correlation matrices,
which can be seen as generalised confusion matrices and reflect the semantic
similarities of the original classes very well for all three exemplary
datasets. The insights gained are valuable and can serve as general label
embedding if a single ground truth per observation cannot be guaranteed.