Kandinsky Conformal Prediction: Efficient Calibration of Image
  Segmentation Algorithms
Joren Brunekreef Eric Marcus Ray Sheombarsing Jan-Jakob Sonke Jonas Teuwen
Image segmentation algorithms can be understood as a collection of pixel
classifiers, for which the outcomes of nearby pixels are correlated. Classifier
models can be calibrated using Inductive Conformal Prediction, but this
requires holding back a sufficiently large calibration dataset for computing
the distribution of non-conformity scores of the model's predictions. If one
only requires only marginal calibration on the image level, this calibration
set consists of all individual pixels in the images available for calibration.
However, if the goal is to attain proper calibration for each individual pixel
classifier, the calibration set consists of individual images. In a scenario
where data are scarce (such as the medical domain), it may not always be
possible to set aside sufficiently many images for this pixel-level
calibration. The method we propose, dubbed ``Kandinsky calibration'', makes use
of the spatial structure present in the distribution of natural images to
simultaneously calibrate the classifiers of ``similar'' pixels. This can be
seen as an intermediate approach between marginal (imagewise) and conditional
(pixelwise) calibration, where non-conformity scores are aggregated over
similar image regions, thereby making more efficient use of the images
available for calibration. We run experiments on segmentation algorithms
trained and calibrated on subsets of the public MS-COCO and Medical Decathlon
datasets, demonstrating that Kandinsky calibration method can significantly
improve the coverage. When compared to both pixelwise and imagewise calibration
on little data, the Kandinsky method achieves much lower coverage errors,
indicating the data efficiency of the Kandinsky calibration.