Assessment of Prediction Intervals Using Uncertainty Characteristics
  Curves
Jiri Navratil Benjamin Elder Matthew Arnold Soumya Ghosh Prasanna Sattigeri
Accurate quantification of model uncertainty has long been recognized as a
fundamental requirement for trusted AI. In regression tasks, uncertainty is
typically quantified using prediction intervals calibrated to an ad-hoc
operating point, making evaluation and comparison across different studies
relatively difficult. Our work leverages: (1) the concept of operating
characteristics curves and (2) the notion of a gain over a null reference, to
derive a novel operating point agnostic assessment methodology for prediction
intervals. The paper defines the Uncertainty Characteristics Curve and
demonstrates its utility in selected scenarios. We argue that the proposed
method addresses the current need for comprehensive assessment of prediction
intervals and thus represents a valuable addition to the uncertainty
quantification toolbox.