Predictive Inference in Multi-environment Scenarios
John C. Duchi Suyash Gupta Kuanhao Jiang Pragya Sur
We address the challenge of constructing valid confidence intervals and sets
in problems of prediction across multiple environments. We investigate two
types of coverage suitable for these problems, extending the jackknife and
split-conformal methods to show how to obtain distribution-free coverage in
such non-traditional, hierarchical data-generating scenarios. Our contributions
also include extensions for settings with non-real-valued responses and a
theory of consistency for predictive inference in these general problems. We
demonstrate a novel resizing method to adapt to problem difficulty, which
applies both to existing approaches for predictive inference with hierarchical
data and the methods we develop; this reduces prediction set sizes using
limited information from the test environment, a key to the methods' practical
performance, which we evaluate through neurochemical sensing and species
classification datasets.