Data-Adaptive Tradeoffs among Multiple Risks in Distribution-Free
  Prediction
Drew T. Nguyen Reese Pathak Anastasios N. Angelopoulos Stephen Bates Michael I. Jordan
Decision-making pipelines are generally characterized by tradeoffs among
various risk functions. It is often desirable to manage such tradeoffs in a
data-adaptive manner. As we demonstrate, if this is done naively, state-of-the
art uncertainty quantification methods can lead to significant violations of
putative risk guarantees.
  To address this issue, we develop methods that permit valid control of risk
when threshold and tradeoff parameters are chosen adaptively. Our methodology
supports monotone and nearly-monotone risks, but otherwise makes no
distributional assumptions.
  To illustrate the benefits of our approach, we carry out numerical
experiments on synthetic data and the large-scale vision dataset MS-COCO.