Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for
  Specialized Tasks
Bálint Mucsányi Michael Kirchhof Seong Joon Oh
Uncertainty quantification, once a singular task, has evolved into a spectrum
of tasks, including abstained prediction, out-of-distribution detection, and
aleatoric uncertainty quantification. The latest goal is disentanglement: the
construction of multiple estimators that are each tailored to one and only one
task. Hence, there is a plethora of recent advances with different intentions -
that often entirely deviate from practical behavior. This paper conducts a
comprehensive evaluation of numerous uncertainty estimators across diverse
tasks on ImageNet. We find that, despite promising theoretical endeavors,
disentanglement is not yet achieved in practice. Additionally, we reveal which
uncertainty estimators excel at which specific tasks, providing insights for
practitioners and guiding future research toward task-centric and disentangled
uncertainty estimation methods. Our code is available at
https://github.com/bmucsanyi/bud.