---
author:
- |
    Meelis Kull\
    Department of Computer Science\
    University of Tartu\
    `meelis.kull@ut.ee`\
    Miquel Perello-Nieto\
    Department of Computer Science\
    University of Bristol\
    `miquel.perellonieto@bris.ac.uk`\
    Markus Kängsepp\
    Department of Computer Science\
    University of Tartu\
    `markus.kangsepp@ut.ee`\
    Telmo Silva Filho\
    Department of Statistics\
    Universidade Federal da Paraíba\
    `telmo@de.ufpb.br`\
    Hao Song\
    Department of Computer Science\
    University of Bristol\
    `hao.song@bristol.ac.uk`\
    Peter Flach\
    Department of Computer Science\
    University of Bristol and\
    The Alan Turing Institute\
    `peter.flach@bristol.ac.uk`\
title: |
    Beyond temperature scaling:\
    Obtaining well-calibrated multi-class probabilities with Dirichlet calibration\
    Supplementary material
---

Source code
===========

The instructions and code for the experiments can be found on .

Proofs
======

\[thm:equiv\] The parametric families $\vmuh_{DirGen}(\vq; \valpha,\vpi)$, $\vmuh_{DirLin}(\vq; \MW,\vb)$ and $\vmuh_{Dir}(\vq; \MA,\vc)$ are equal, i.e. they contain exactly the same calibration maps.

We will prove that:

1.  every function in $\vmuh_{DirGen}(\vq; \valpha,\vpi)$ belongs to $\vmuh_{DirLin}(\vq; \MW,\vb)$;

2.  every function in $\vmuh_{DirLin}(\vq; \MW,\vb)$ belongs to $\vmuh_{Dir}(\vq; \MA,\vc)$;

3.  every function in $\vmuh_{Dir}(\vq; \MA,\vc)$ belongs to $\vmuh_{DirGen}(\vq; \valpha,\vpi)$.

#### 1.

Consider a function $\muh(\vq)=\vmuh_{DirGen}(\vq; \valpha,\vpi)$. Let us start with an observation that any vector $\vx=(x_1,\dots,x_k)\in(0,\infty)^k$ with only positive elements can be renormalised to add up to $1$ using the expression $\softmax(\vln(\vx))$, since: $$\begin{aligned}
\softmax(\vln(\vx))=\vexp(\vln(\vx))/(\sum_i \exp(\ln(x_i)))=\vx/(\sum_i x_i)\end{aligned}$$ where $\vexp$ is an operator applying exponentiation element-wise. Therefore, $$\begin{aligned}
    \muh(\vq)=\softmax(\vln(\pi_1 f_1(\vq),\dots,\pi_k f_k(\vq)))
%    =\softmax(\vln\pi+\vln(f_1(\vq),\dots,f_k(\vq)))\end{aligned}$$ where $f_i(\vq)$ is the probability density function of the distribution $Dir(\valpha^{(i)})$ where $\valpha^{(i)}$ is the $i$-th row of matrix $\valpha$. Hence, $f_i(\vq)=\frac{1}{B(\valpha^{(i)})}\prod_{j=1}^k q_j^{\alpha_{ij}-1}$, where $B(\cdot)$ denotes the multivariate beta function. Let us define a matrix $\MW$ and vector $\vb$ as follows: $$\begin{aligned}
w_{ij}=\alpha_{ij}-1,\qquad b_i=\ln(\pi_i)-\ln(B(\valpha^{(i)}))\end{aligned}$$ with $w_{ij}$ and $\alpha_{ij}$ denoting elements of matrices $\MW$ and $\valpha$, respectively, and $b_i,\pi_i$ denoting elements of vectors $\vb$ and $\vpi$. Now we can write $$\begin{aligned}
   \ln(\pi_i f_i(\vq))
   &=\ln(\pi_i)-\ln(B(\valpha^{(i)}))+\ln\prod_{j=1}^k q_j^{\alpha_{ij}-1} \\
   &=\ln(\pi_i)-\ln(B(\valpha^{(i)}))+\sum_{j=1}^k (\alpha_{ij}-1)\ln(q_j) \\
   &=b_i+\sum_{j=1}^k w_{ij}\ln(q_j)\end{aligned}$$ and substituting this back into $\muh(\vq)$ we get: $$\begin{aligned}
\muh(\vq)&=\softmax(\vln(\pi_1 f_1(\vq),\dots,\pi_k f_k(\vq))) \\
&=\softmax(\vb+\MW\vln(\vq))=\muh_{DirLin}(\vq; \MW,\vb)\end{aligned}$$

#### 2.

Consider a function $\muh(\vq)=\vmuh_{DirLin}(\vq; \MW,\vb)$. Let us define a matrix $\MA$ and vector $\vc$ as follows: $$\begin{aligned}
a_{ij}=w_{ij}-\min_{i}w_{ij},\qquad \vc=\softmax(\MW\,\vln\,\vu+\vb)\end{aligned}$$ with $a_{ij}$ and $w_{ij}$ denoting elements of matrices $\MA$ and $\MW$, respectively, and $\vu=(1/k,\dots,1/k)$ is a column vector of length $k$. Note that $\MA\,\vx=\MW\,\vx+const_1$ and $\vln\,\softmax(\vx)=\vx+const_2$ for any $x$ where $const_1$ and $const_2$ are constant vectors (all elements are equal), but the constant depends on $\vx$. Taking into account that $\softmax(\vv+const)=\softmax(\vv)$ for any vector $\vv$ and constant vector $const$, we obtain: $$\begin{aligned}
\muh_{Dir}(\vq; \MA,\vc)
&=\softmax(\MA\,\vln\,\frac{\vq}{1/k}+\vln\,\vc)
=\softmax(\MW\,\vln\,\frac{\vq}{1/k}+const_1+\vln\,\vc) \\
&=\softmax(\MW\,\vln\,\vq-\MW\,\vln\,\vu+const_1+\vln\,\softmax(\MW\,\vln\,\vu+\vb)) \\
&=\softmax(\MW\,\vln\,\vq-\MW\,\vln\,\vu+const_1+\MW\,\vln\,\vu+\vb+const_2) \\
&=\softmax(\MW\,\vln\,\vq+\vb+const_1+const_2)
=\softmax(\MW\,\vln\,\vq+\vb)=\muh_{DirLin}(\vq; \MW,\vb)\\
&=\muh(\vq)\end{aligned}$$

#### 3.

Consider a function $\muh(\vq)=\vmuh_{Dir}(\vq; \MA,\vc)$. Let us define a matrix $\valpha$, vector $\vb$ and vector $\pi$ as follows: $$\begin{aligned}
\alpha_{ij}=a_{ij}+1,\qquad \vb=\vln\,\vc-\MA\,\vln\,\vu,\qquad \pi_i=\exp(b_i)\cdot B(\valpha^{(i)})\end{aligned}$$ with $\alpha_{ij}$ and $a_{ij}$ denoting elements of matrices $\valpha$ and $\MA$, respectively, and $\vu=(1/k,\dots,1/k)$ is a column vector of length $k$. We can now write: $$\begin{aligned}
\muh(\vq)&=\muh_{Dir}(\vq; \MA,\vc)
=\softmax(\MA\,\vln\,\frac{\vq}{1/k}+\vln\,\vc)
=\softmax(\MA\,\vln\,\vq-\MA\,\vln\,\vu+\vln\,\vc) \\
&=\softmax((\valpha-1)\vln\,\vq+\vb)\end{aligned}$$ Element $i$ in the vector within the softmax is equal to: $$\begin{aligned}
    \sum_{j=1}^k (\valpha_{ij}-1)\ln(q_j)+b_j 
    &= \sum_{j=1}^k (\valpha_{ij}-1)\ln(q_j) +\ln(\pi_i\cdot\frac{1}{B(\valpha^{(i)})}) \\
    &= \ln(\pi_i\cdot \frac{1}{B(\valpha^{(i)})} \prod_{j=1}^k q_j^{\valpha_{ij}-1}) \\
    &= \ln(\pi_i\cdot f_i(\valpha^{(i)}))\end{aligned}$$ and therefore: $$\begin{aligned}
\muh(\vq)=\softmax((\valpha-1)\vln(\vq)+\vb)=\softmax(\ln(\pi_i\cdot f_i(\valpha^{(i)})))=\vmuh_{DirGen}(\vq; \valpha,\vpi)\end{aligned}$$

The following proposition proves that temperature scaling can be viewed as a general-purpose calibration method, being a special case within the Dirichlet calibration map family.

Let us denote the temperature scaling family by $\muh'_{TempS}(\vz; t)=\softmax(\vz/t)$ where $\vz$ are the logits. Then for any $t$, temperature scaling can be expressed as $$\begin{aligned}
\muh'_{TempS}(\vz; t)=\muh_{DirLin}(\softmax(\vz); \frac{1}{t}\MI, \vzero)\end{aligned}$$ where $\MI$ is the identity matrix and $\vzero$ is the vector of zeros.

Let us first observe that for any $\vx\in\sR^k$ there exists a constant vector $const$ (all elements are equal) such that $\vln\,\softmax(\vx)=\vx+const$. Furthermore, $\softmax(\vv+const)=\softmax(\vv)$ for any vector $\vv$ and any constant vector $const$. Therefore, $$\begin{aligned}
\muh_{DirLin}(\softmax(\vz); \frac{1}{t}\MI, \vzero)
&=\softmax(\frac{1}{t}\,\MI\,\vln\,\softmax(\vz))) \\
&=\softmax(\frac{1}{t}\,\MI\,(\vz+const)) \\
&=\softmax(\frac{1}{t}\,\MI\,\vz+\frac{1}{t}\,\MI\, const) \\
&=\softmax(\vz/t+const') \\
&=\softmax(\vz/t) \\
&=\muh'_{TempS}(\vz; t)\end{aligned}$$ where $const'=\frac{1}{t}\,\MI\, const$ is a constant vector as a product of a diagonal matrix with a constant vector.

Dirichlet calibration
=====================

In this section we show some examples of reliability diagrams and other plots that can help to understand the representational power of Dirichlet calibration compared with other calibration methods.

Reliability diagram examples
----------------------------

We will look at two examples of reliability diagrams on the original classifier and after applying $6$ calibration methods. Figure \[fig:mlp:bs:reldiag\] shows the first example for the 3 class classification dataset *balance-scale* and the classifier MLP. This figure shows the confidence-reliability diagram in the first column and the classwise-reliability diagrams in the other columns. Figure \[fig:nb:reldiag:mlp:bal:uncal\] shows how posterior probabilities from the MLP have small gaps between the true class proportions and the predicted means. This visualisation may indicate that the original classifier is already well calibrated. However, when we separate the reliability diagram per class, we notice that the predictions for the first class are underconfident; as indicated by low mean predictions containing high proportions of the true class. On the other hand, classes 2 and 3 are overconfident in the regions of posterior probabilities compressed between $[0.2, 0.5]$ while being underconfident in higher regions. The discrepancy shown by analysing the individual reliability diagrams seems to compensate in the general picture of the aggregated one.

The following subfigures show how the different calibration methods try to reduce ECE, occasionally increasing the error. As can be seen in Table \[table:mlp:balance:ece\], Dirichlet L2 and One-vs.Rest isotonic regression obtain the lowest ECE while One-vs.Rest frequency binning makes the original calibration worse. Looking at Figure \[fig:nb:reldiag:mlp:bal:temp\] it is possible to see how temperature scaling manages to reduce the overall overconfidence in the higher range of probabilities for classes 2 and 3, but makes the situation worse in the interval $[0.2, 0.6]$. However, it manages to reduce the overall ECE.

[.26]{} ![Confidence-reliability diagrams in the first column and classwise-reliability diagrams in the remaining columns, for a real experiment with the multilayer perceptron classifier on the balance-scale dataset and a subset of the calibrators. All the test partitions from the 5 times 5-fold-cross-validation have been aggregated to draw every plot.[]{data-label="fig:mlp:bs:reldiag"}](figures/results/mlp_balance-scale_uncalibrated_conf_rel_diagr.pdf "fig:"){width="\linewidth"}

[.71]{} ![Confidence-reliability diagrams in the first column and classwise-reliability diagrams in the remaining columns, for a real experiment with the multilayer perceptron classifier on the balance-scale dataset and a subset of the calibrators. All the test partitions from the 5 times 5-fold-cross-validation have been aggregated to draw every plot.[]{data-label="fig:mlp:bs:reldiag"}](figures/results/mlp_balance-scale_uncalibrated_rel_diagr_perclass.pdf "fig:"){width="\linewidth"}

[.26]{} ![Confidence-reliability diagrams in the first column and classwise-reliability diagrams in the remaining columns, for a real experiment with the multilayer perceptron classifier on the balance-scale dataset and a subset of the calibrators. All the test partitions from the 5 times 5-fold-cross-validation have been aggregated to draw every plot.[]{data-label="fig:mlp:bs:reldiag"}](figures/results/mlp_balance-scale_binning_freq_conf_rel_diagr.pdf "fig:"){width="\linewidth"}

[.71]{} ![Confidence-reliability diagrams in the first column and classwise-reliability diagrams in the remaining columns, for a real experiment with the multilayer perceptron classifier on the balance-scale dataset and a subset of the calibrators. All the test partitions from the 5 times 5-fold-cross-validation have been aggregated to draw every plot.[]{data-label="fig:mlp:bs:reldiag"}](figures/results/mlp_balance-scale_binning_freq_rel_diagr_perclass.pdf "fig:"){width="\linewidth"}

[.26]{} ![Confidence-reliability diagrams in the first column and classwise-reliability diagrams in the remaining columns, for a real experiment with the multilayer perceptron classifier on the balance-scale dataset and a subset of the calibrators. All the test partitions from the 5 times 5-fold-cross-validation have been aggregated to draw every plot.[]{data-label="fig:mlp:bs:reldiag"}](figures/results/mlp_balance-scale_binning_width_conf_rel_diagr.pdf "fig:"){width="\linewidth"}

[.71]{} ![Confidence-reliability diagrams in the first column and classwise-reliability diagrams in the remaining columns, for a real experiment with the multilayer perceptron classifier on the balance-scale dataset and a subset of the calibrators. All the test partitions from the 5 times 5-fold-cross-validation have been aggregated to draw every plot.[]{data-label="fig:mlp:bs:reldiag"}](figures/results/mlp_balance-scale_binning_width_rel_diagr_perclass.pdf "fig:"){width="\linewidth"}

[.26]{} ![Confidence-reliability diagrams in the first column and classwise-reliability diagrams in the remaining columns, for a real experiment with the multilayer perceptron classifier on the balance-scale dataset and a subset of the calibrators. All the test partitions from the 5 times 5-fold-cross-validation have been aggregated to draw every plot.[]{data-label="fig:mlp:bs:reldiag"}](figures/results/mlp_balance-scale_isotonic_conf_rel_diagr.pdf "fig:"){width="\linewidth"}

[.71]{} ![Confidence-reliability diagrams in the first column and classwise-reliability diagrams in the remaining columns, for a real experiment with the multilayer perceptron classifier on the balance-scale dataset and a subset of the calibrators. All the test partitions from the 5 times 5-fold-cross-validation have been aggregated to draw every plot.[]{data-label="fig:mlp:bs:reldiag"}](figures/results/mlp_balance-scale_isotonic_rel_diagr_perclass.pdf "fig:"){width="\linewidth"}

[.26]{} ![Confidence-reliability diagrams in the first column and classwise-reliability diagrams in the remaining columns, for a real experiment with the multilayer perceptron classifier on the balance-scale dataset and a subset of the calibrators. All the test partitions from the 5 times 5-fold-cross-validation have been aggregated to draw every plot.[]{data-label="fig:mlp:bs:reldiag"}](figures/results/mlp_balance-scale_dirichlet_fix_diag_conf_rel_diagr.pdf "fig:"){width="\linewidth"}

[.71]{} ![Confidence-reliability diagrams in the first column and classwise-reliability diagrams in the remaining columns, for a real experiment with the multilayer perceptron classifier on the balance-scale dataset and a subset of the calibrators. All the test partitions from the 5 times 5-fold-cross-validation have been aggregated to draw every plot.[]{data-label="fig:mlp:bs:reldiag"}](figures/results/mlp_balance-scale_dirichlet_fix_diag_rel_diagr_perclass.pdf "fig:"){width="\linewidth"}

[.26]{} ![Confidence-reliability diagrams in the first column and classwise-reliability diagrams in the remaining columns, for a real experiment with the multilayer perceptron classifier on the balance-scale dataset and a subset of the calibrators. All the test partitions from the 5 times 5-fold-cross-validation have been aggregated to draw every plot.[]{data-label="fig:mlp:bs:reldiag"}](figures/results/mlp_balance-scale_ovr_dir_full_conf_rel_diagr.pdf "fig:"){width="\linewidth"}

[.71]{} ![Confidence-reliability diagrams in the first column and classwise-reliability diagrams in the remaining columns, for a real experiment with the multilayer perceptron classifier on the balance-scale dataset and a subset of the calibrators. All the test partitions from the 5 times 5-fold-cross-validation have been aggregated to draw every plot.[]{data-label="fig:mlp:bs:reldiag"}](figures/results/mlp_balance-scale_ovr_dir_full_rel_diagr_perclass.pdf "fig:"){width="\linewidth"}

[.26]{} ![Confidence-reliability diagrams in the first column and classwise-reliability diagrams in the remaining columns, for a real experiment with the multilayer perceptron classifier on the balance-scale dataset and a subset of the calibrators. All the test partitions from the 5 times 5-fold-cross-validation have been aggregated to draw every plot.[]{data-label="fig:mlp:bs:reldiag"}](figures/results/mlp_balance-scale_dirichlet_full_l2_conf_rel_diagr.pdf "fig:"){width="\linewidth"}

[.71]{} ![Confidence-reliability diagrams in the first column and classwise-reliability diagrams in the remaining columns, for a real experiment with the multilayer perceptron classifier on the balance-scale dataset and a subset of the calibrators. All the test partitions from the 5 times 5-fold-cross-validation have been aggregated to draw every plot.[]{data-label="fig:mlp:bs:reldiag"}](figures/results/mlp_balance-scale_dirichlet_full_l2_rel_diagr_perclass.pdf "fig:"){width="\linewidth"}

In the second example we show $3$ calibration methods for a 4 class classification problem (car dataset) applied on the scores of an Adaboost SAMME classifier. Figure \[fig:nb:reldiag:adas:car\] shows one reliability diagram per class ($C_1$ *acceptable*, $C_2$ *good*, $C_3$ *unacceptable*, and $C_4$ *very good*).

From this Figure we can see that the uncalibrated model is underconfident for classes 1, 2 and 3, showing posterior probabilities never higher than $0.7$, while having true class proportions higher than $0.7$ in the mentioned interval. We can see that after applying some of the calibration models the posterior probabilities reach higher probability values.

As can be seen in Table \[table:adas:car:ece\], Dirichlet L2 and One-vs.Rest Isotonic Regression obtain the lowest ECE while Temperature Scaling makes the original calibration worse. Figure \[fig:nb:reldiag:class:adas:car:dirl2\] shows how Dirichlet calibration with L2 regularisation achieved the largest spread of probabilities, also reducing the error mean gap with the predictions and the true class proportions. On the other hand, temperature scaling reduced ECE for class 1, but hurt the overall performance for the other classes.

[.9]{} ![Reliability diagrams per class for a real experiment with the classifier Ada boost SAMME on the car dataset and $3$ calibrators. The test partitions from the 5 times 5-fold-cross-validation have been aggregated to draw every plot.[]{data-label="fig:nb:reldiag:adas:car"}](figures/results/adas_car_uncalibrated_rel_diagr_perclass.pdf "fig:"){width="\linewidth"}

[.9]{} ![Reliability diagrams per class for a real experiment with the classifier Ada boost SAMME on the car dataset and $3$ calibrators. The test partitions from the 5 times 5-fold-cross-validation have been aggregated to draw every plot.[]{data-label="fig:nb:reldiag:adas:car"}](figures/results/adas_car_isotonic_rel_diagr_perclass.pdf "fig:"){width="\linewidth"}

[.9]{} ![Reliability diagrams per class for a real experiment with the classifier Ada boost SAMME on the car dataset and $3$ calibrators. The test partitions from the 5 times 5-fold-cross-validation have been aggregated to draw every plot.[]{data-label="fig:nb:reldiag:adas:car"}](figures/results/adas_car_dirichlet_fix_diag_rel_diagr_perclass.pdf "fig:"){width="\linewidth"}

[.9]{} ![Reliability diagrams per class for a real experiment with the classifier Ada boost SAMME on the car dataset and $3$ calibrators. The test partitions from the 5 times 5-fold-cross-validation have been aggregated to draw every plot.[]{data-label="fig:nb:reldiag:adas:car"}](figures/results/adas_car_dirichlet_full_l2_rel_diagr_perclass.pdf "fig:"){width="\linewidth"}

A more detailed depiction of the previous reliability diagrams can be seen in Figure \[fig:nb:pos:scores:class\]. In this case, the posterior probabilities are not introduced in bins, but a boxplot summarises their full distribution. The first observation here is, for the *good* and *very good* classes, the uncalibrated model tends to predict probability vectors with small variance, i.e. the outputs do not change much among different instances. Among the calibration approaches, temperature scaling still maintains this low level of variance, while both isotonic and Dirichlet L2 manage to show a higher variance on the outputs. While this observation cannot be justified here without quantitative analysis, another observation clearly shows an advantage of using Dirichlet L2. For the *acceptable* class, only Dirichlet L2 is capable of providing the highest mean probability for the correct class, while the other three methods tend to put higher probability mass on the *unacceptable* class on average.

[.24]{} ![Effect of Dirichlet Calibration on the scores of Ada boost SAMME on the *car* dataset which is composed of $4$ classes (*acceptable*, *good*, *unacceptable*, and *very good*). The whiskers of each box indicate the 5th and 95th percentile, the notch around the median indicates the confidence interval. The [green]{} error bar to the right of each box indicates one standard deviation on each side of the mean. In each subfigure, the first boxplot corresponds to the posterior probabilities for the samples of class 1, divided in 4 boxes representing the posterior probabilities for each class. A good classifier should have the highest posterior probabilities in the box corresponding to the true class. In Figure \[fig:nb:pos:scores:class:adas:car:uncal\] it is possible to see that the first class (*acceptable*) is missclassified as belonging to the third class (*unacceptable*) with high probability values, while Dirichlet Calibration is able to alleviate that problem. Also, for the second and fourth true classes (*good*, and *very good*) the original classifier uses a reduced domain of probabilities (indicative of underconfidence), while Dirichlet calibration is able to spread these probabilities with more meaningful values (as indicated by a reduction of the calibration losses; See Figure \[fig:nb:reldiag:adas:car\]). []{data-label="fig:nb:pos:scores:class"}](figures/results/adas_car_uncalibrated_positive_scores_per_class.pdf "fig:"){width="\linewidth"}

[.24]{} ![Effect of Dirichlet Calibration on the scores of Ada boost SAMME on the *car* dataset which is composed of $4$ classes (*acceptable*, *good*, *unacceptable*, and *very good*). The whiskers of each box indicate the 5th and 95th percentile, the notch around the median indicates the confidence interval. The [green]{} error bar to the right of each box indicates one standard deviation on each side of the mean. In each subfigure, the first boxplot corresponds to the posterior probabilities for the samples of class 1, divided in 4 boxes representing the posterior probabilities for each class. A good classifier should have the highest posterior probabilities in the box corresponding to the true class. In Figure \[fig:nb:pos:scores:class:adas:car:uncal\] it is possible to see that the first class (*acceptable*) is missclassified as belonging to the third class (*unacceptable*) with high probability values, while Dirichlet Calibration is able to alleviate that problem. Also, for the second and fourth true classes (*good*, and *very good*) the original classifier uses a reduced domain of probabilities (indicative of underconfidence), while Dirichlet calibration is able to spread these probabilities with more meaningful values (as indicated by a reduction of the calibration losses; See Figure \[fig:nb:reldiag:adas:car\]). []{data-label="fig:nb:pos:scores:class"}](figures/results/adas_car_isotonic_positive_scores_per_class.pdf "fig:"){width="\linewidth"}

[.24]{} ![Effect of Dirichlet Calibration on the scores of Ada boost SAMME on the *car* dataset which is composed of $4$ classes (*acceptable*, *good*, *unacceptable*, and *very good*). The whiskers of each box indicate the 5th and 95th percentile, the notch around the median indicates the confidence interval. The [green]{} error bar to the right of each box indicates one standard deviation on each side of the mean. In each subfigure, the first boxplot corresponds to the posterior probabilities for the samples of class 1, divided in 4 boxes representing the posterior probabilities for each class. A good classifier should have the highest posterior probabilities in the box corresponding to the true class. In Figure \[fig:nb:pos:scores:class:adas:car:uncal\] it is possible to see that the first class (*acceptable*) is missclassified as belonging to the third class (*unacceptable*) with high probability values, while Dirichlet Calibration is able to alleviate that problem. Also, for the second and fourth true classes (*good*, and *very good*) the original classifier uses a reduced domain of probabilities (indicative of underconfidence), while Dirichlet calibration is able to spread these probabilities with more meaningful values (as indicated by a reduction of the calibration losses; See Figure \[fig:nb:reldiag:adas:car\]). []{data-label="fig:nb:pos:scores:class"}](figures/results/adas_car_dirichlet_fix_diag_positive_scores_per_class.pdf "fig:"){width="\linewidth"}

[.24]{} ![Effect of Dirichlet Calibration on the scores of Ada boost SAMME on the *car* dataset which is composed of $4$ classes (*acceptable*, *good*, *unacceptable*, and *very good*). The whiskers of each box indicate the 5th and 95th percentile, the notch around the median indicates the confidence interval. The [green]{} error bar to the right of each box indicates one standard deviation on each side of the mean. In each subfigure, the first boxplot corresponds to the posterior probabilities for the samples of class 1, divided in 4 boxes representing the posterior probabilities for each class. A good classifier should have the highest posterior probabilities in the box corresponding to the true class. In Figure \[fig:nb:pos:scores:class:adas:car:uncal\] it is possible to see that the first class (*acceptable*) is missclassified as belonging to the third class (*unacceptable*) with high probability values, while Dirichlet Calibration is able to alleviate that problem. Also, for the second and fourth true classes (*good*, and *very good*) the original classifier uses a reduced domain of probabilities (indicative of underconfidence), while Dirichlet calibration is able to spread these probabilities with more meaningful values (as indicated by a reduction of the calibration losses; See Figure \[fig:nb:reldiag:adas:car\]). []{data-label="fig:nb:pos:scores:class"}](figures/results/adas_car_dirichlet_full_l2_positive_scores_per_class.pdf "fig:"){width="\linewidth"}

Experimental setup {#sec:exp}
==================

In this section we provide the detailed description of the experimental setup on a variety of non-neural classifiers and datasets. While our implementation of Dirichlet calibration is based on standard Newton-Raphson with multinomial logistic loss and L2 regularisation, as mentioned at the end of Section 3, existing implementations of logistic regression (e.g. scikit-learn) with the log transformed predicted probabilities can also be easily applied.

Datasets and performance estimation
-----------------------------------

The full list of datasets, and a brief description of each one including the number of samples, features and classes is presented in Table \[tab:data\].

Figure \[fig:ds:partition\] shows how every dataset was divided in order to get an estimated performance for every combination of dataset, classifier and calibrator. Each dataset was divided using 5 times 5-fold-cross-validation to create 25 test partitions. For each of the 25 partitions the corresponding training set was divided further with a 3-fold-cross-validation for wich the bigger portions were used to train the classifiers (and validate the calibratiors if they had hyperparameters), and the small portion was used to train the calibrators. The 3 calibrators trained in the inner 3-folds were used to predict the corresponding test partition, and their predictions were averaged in order to obtain better estimates of their performance with the 7 different metrics (accuracy, Brier score, log-loss, maximum calibration error, confidence-ECE, classwise-ECE and the p test statistic of the ECE metrics). Finally, the 25 resulting measures were averaged.

\[tab:data\]

![image](figures/experiments/datasets_train_test_partitions.pdf){width="0.9\linewidth"} \[fig:ds:partition\]

Full example of statistical analysis {#sec:exp:example}
------------------------------------

The following is a full example of how the final rankings and statistical tests are computed. For this example, we will focus on the metric log-loss, and we will start with the naive Bayes classifier. Table \[table:nbayes:loss\] shows the estimated log-loss by averaging the 5-times 5-fold cross-validation log-losses of the inner 3-fold aggregated predictions. The sub-indices are the ranking of every calibrator for each dataset (ties in the ranking share the averaged rank). The resulting table of sub-indices is used to compute the Friedman test statistic, resulting in a value of $73.8$ and a p-value of $6.71e^{-14}$ indicating statistical difference between the calibration methods. The last row contains the average ranks of the full table, which is shown in the corresponding critical difference diagram in Figure \[fig:cd:nbayes:loss\]. The critical difference uses the Bonferroni-Dunn one-tailed statistical test to compute the minimum ranking distance that is shown in the Figure, indicating that for this particular classifier and metric the Dirichlet calibrator with L2 regularisation is significantly better than the other methods.

[.49]{} ![Critical Difference diagrams for the averaged ranking results of the metric Log-loss.](figures/results/crit_diff_nbayes_loss.pdf "fig:"){width="\linewidth"}

[.49]{} ![Critical Difference diagrams for the averaged ranking results of the metric Log-loss.](figures/results/crit_diff_loss_v2 "fig:"){width="\linewidth"}

The same process is applied to each of the $11$ classifiers for every metric. Table \[table:loss\] shows the final average results of all classifiers. Notice that the row corresponding to naive Bayes has the rounded average rankings from Figure \[fig:cd:nbayes:loss\].

Results {#sec:res}
=======

In this Section we present all the final results, including ranking tables for every metric, critical difference diagrams, the best hyperparameters selected for Dirichlet calibration with L2 regularisation, Frequency binning and Width binning; a comparison of how calibrated the $11$ classifiers are, and additional results on deep neural networks.

Final ranking tables for all metrics {#sec:res:rank}
------------------------------------

We present here all the final ranking tables for all metrics (Tables \[table:acc\], \[table:loss\], \[table:brier\], \[table:mce\], \[table:conf-ece\], \[table:cw-ece\], \[table:p-conf-ece\], and \[table:p-cw-ece\]). For each ranking, a lower value is indicative of a better metric value (eg. a higher accuracy corresponds to a lower ranking, while a lower log-loss corresponds to a lower ranking as well). Additional details on how to interpret the tables can be found in Section \[sec:exp:example\].

Final critical difference diagrams for every metric
---------------------------------------------------

In order to perform a final comparison between calibration methods, we considered every combination of dataset and classifier as a group $n = \#datasets \times \#classifiers$, and ranked the results of the $k$ calibration methods. With this setting, we have performed the Friedman statistical test followed by the one-tailed Bonferroni-Dunn test to obtain critical differences (CDs) for every metric (See Figure \[fig:multi:cd:all\]). The results showed Dirichlet L2 as the best calibration method for the measures accuracy, log-loss and p-cw-ece with statistical significance (See Figures \[fig:multi:cd:acc\] \[fig:multi:cd:logloss\], and \[fig:multi:cd:p-cw-ece\]), and in the group of the best calibration methods in the rest of the metrics with statistical significance, but no difference within the group. It is worth mentioning that Figure \[fig:multi:cd:logloss\] showed statistical difference between Dirichlet L2, OvR Beta, OvR width binning, and the rest of the calibrators in one group; in the mentioned order.

[.49]{} ![Critical difference of the average of multiclass classifiers.[]{data-label="fig:multi:cd:all"}](figures/results/crit_diff_acc_v2 "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of the average of multiclass classifiers.[]{data-label="fig:multi:cd:all"}](figures/results/crit_diff_brier_v2 "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of the average of multiclass classifiers.[]{data-label="fig:multi:cd:all"}](figures/results/crit_diff_loss_v2 "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of the average of multiclass classifiers.[]{data-label="fig:multi:cd:all"}](figures/results/crit_diff_mce_v2 "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of the average of multiclass classifiers.[]{data-label="fig:multi:cd:all"}](figures/results/crit_diff_conf-ece_v2.pdf "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of the average of multiclass classifiers.[]{data-label="fig:multi:cd:all"}](figures/results/crit_diff_cw-ece_v2.pdf "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of the average of multiclass classifiers.[]{data-label="fig:multi:cd:all"}](figures/results/crit_diff_p-conf-ece_v2.pdf "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of the average of multiclass classifiers.[]{data-label="fig:multi:cd:all"}](figures/results/crit_diff_p-cw-ece_v2.pdf "fig:"){width="\linewidth"}

[.4]{} ![Proportion of times each calibrator passes a calibration p-test with a p-value higher than 0.05.[]{data-label="fig:cal:p:ece"}](figures/results/p_table_calibrators_p-conf-ece.pdf "fig:"){width="\linewidth"}

[.4]{} ![Proportion of times each calibrator passes a calibration p-test with a p-value higher than 0.05.[]{data-label="fig:cal:p:ece"}](figures/results/p_table_calibrators_p-cw-ece.pdf "fig:"){width="\linewidth"}

Best calibrator hyperparameters
-------------------------------

[.30]{} ![Histogram of the selected hyperparameters during the inner 3-fold-cross-validation[]{data-label="fig:hyper"}](figures/results/bars_hyperparameters_all_Dirichlet_L2.pdf "fig:"){width="\linewidth"}

[.30]{} ![Histogram of the selected hyperparameters during the inner 3-fold-cross-validation[]{data-label="fig:hyper"}](figures/results/bars_hyperparameters_all_OvR_Freq_Bin.pdf "fig:"){width="\linewidth"}

[.30]{} ![Histogram of the selected hyperparameters during the inner 3-fold-cross-validation[]{data-label="fig:hyper"}](figures/results/bars_hyperparameters_all_OvR_Width_Bin.pdf "fig:"){width="\linewidth"}

Figure \[fig:hyper\] shows the best hyperparameters for every inner 3-fold-cross-validation. Dirichlet L2 (Figure \[fig:hyper:dir:l2\]) shows a preference for regularisation hyperparameter $\lambda = 1e^{-3}$ and lower values. Our current minimum regularisation value of $1e^{-7}$ is also being selected multiple times, indicating that lower values may be optimal in several occasions. However, this fact did not seem to hurt the overall good results in our experiments. One-vs.-Rest frequency binning tends to prefer $10$ bins of equal number of samples, while One-vs.Rest width binning prefers $5$ equal sized bins (See Figures \[fig:hyper:freq:bin\] and \[fig:hyper:width:bin\] respectively).

Comparison of classifiers
-------------------------

In this Section we compare all the classifiers without post-hoc calibration on $17$ of the datasets; from the total of $21$ datasets *shuttle*, *yeast*, *mfeat-karhunen* and *libras-movement* were removed from this analysis as at least one classifier was not able to complete the experiment.

[.49]{} ![Critical difference of uncalibrated classifiers.[]{data-label="fig:uncal:cd:all"}](figures/results/crit_diff_uncal_classifiers_acc "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of uncalibrated classifiers.[]{data-label="fig:uncal:cd:all"}](figures/results/crit_diff_uncal_classifiers_loss "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of uncalibrated classifiers.[]{data-label="fig:uncal:cd:all"}](figures/results/crit_diff_uncal_classifiers_brier "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of uncalibrated classifiers.[]{data-label="fig:uncal:cd:all"}](figures/results/crit_diff_uncal_classifiers_mce "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of uncalibrated classifiers.[]{data-label="fig:uncal:cd:all"}](figures/results/crit_diff_uncal_classifiers_conf-ece "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of uncalibrated classifiers.[]{data-label="fig:uncal:cd:all"}](figures/results/crit_diff_uncal_classifiers_cw-ece "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of uncalibrated classifiers.[]{data-label="fig:uncal:cd:all"}](figures/results/crit_diff_uncal_classifiers_p-conf-ece "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of uncalibrated classifiers.[]{data-label="fig:uncal:cd:all"}](figures/results/crit_diff_uncal_classifiers_p-cw-ece "fig:"){width="\linewidth"}

Figure \[fig:uncal:cd:all\] shows the Critical Difference diagram for all the $8$ metrics. In particular, the MLP and the SVC with linear kernel are always in the group with the higher rankings and never in the lowest. Similarly, random forest is consistently in the best group, but in the worst group as well in $4$ of the measures. SVC with radial basis kernel is in the best group $6$ times, but $3$ times in the worst. On the other hand, naive Bayes and Adaboost SAMME are consistently in the worst group and never in the best one. The rest of the classifiers did not show a clear ranking position.

[.4]{} ![Proportion of times each classifier is already calibrated with different p-tests.[]{data-label="fig:uncal:p:ece"}](figures/results/p_table_classifiers_p-conf-ece.pdf "fig:"){width="\linewidth"}

[.4]{} ![Proportion of times each classifier is already calibrated with different p-tests.[]{data-label="fig:uncal:p:ece"}](figures/results/p_table_classifiers_p-cw-ece.pdf "fig:"){width="\linewidth"}

Figures \[fig:uncal:p:cw:ece\] and \[fig:uncal:p:conf:ece\] show the proportion of times each classifier passed the p-conf-ECE and p-cw-ECE statistical test for all datasets and cross-validation folds.

Deep neural networks
--------------------

In this section, we provide further discussion about results from the deep networks experiments. These are given in the form of critical difference diagrams (Figure \[fig:dnn:cd:all\]) and tables (Tables \[table:res:dnn:loss\]-\[table:res:dnn:pece\_cw\]) both including the following measures: error rate, log-loss, Brier score, maximum calibration error (MCE), confidence-ECE (conf-ECE), classwise-ECE (cw-ECE), as well as significance measures p-conf-ECE and p-cw-ECE.

In addition, Table \[table:res:dnn:ms\_vs\_vecs\] compares MS-ODIR and vector scaling on log-loss. On the table, we also added MS-ODIR-zero which was obtained from the respective MS-ODIR model by replacing the off-diagonal entries with zeroes. Each experiment is replicated three times with different splits on datasets. This is done to compare the stability of the methods. In each replication, the best scoring model is written in bold.

Finally, Figure \[fig:res:rd\_ece:class4\] shows that temperature scaling systematically under-estimates class 4 probabilities on the model [c10\_resnet\_wide32]{} on CIFAR-10.

[.49]{} ![Critical difference of the deep neural networks.[]{data-label="fig:dnn:cd:all"}](figures/cd_diag_dnn/crit_diff_Error_v2.pdf "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of the deep neural networks.[]{data-label="fig:dnn:cd:all"}](figures/cd_diag_dnn/crit_diff_Brier_v2.pdf "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of the deep neural networks.[]{data-label="fig:dnn:cd:all"}](figures/cd_diag_dnn/crit_diff_Loss_v2.pdf "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of the deep neural networks.[]{data-label="fig:dnn:cd:all"}](figures/cd_diag_dnn/crit_diff_MCE_v2.pdf "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of the deep neural networks.[]{data-label="fig:dnn:cd:all"}](figures/cd_diag_dnn/crit_diff_ECE_v2.pdf "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of the deep neural networks.[]{data-label="fig:dnn:cd:all"}](figures/cd_diag_dnn/crit_diff_ECE_CW_v2.pdf "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of the deep neural networks.[]{data-label="fig:dnn:cd:all"}](figures/cd_diag_dnn/crit_diff_pECE_v2.pdf "fig:"){width="\linewidth"}

[.49]{} ![Critical difference of the deep neural networks.[]{data-label="fig:dnn:cd:all"}](figures/cd_diag_dnn/crit_diff_pECE_cw_v2.pdf "fig:"){width="\linewidth"}

![Reliability diagrams of c10\_resnet\_wide32 on CIFAR-10: (a) classwise-reliability for class 4 after temperature scaling; (b) classwise-reliability for class 4 after Dirichlet calibration.[]{data-label="fig:res:rd_ece:class4"}](figures/results/figure_RD_ECE_class4.pdf){width="\linewidth"}
