Criterion collapse and loss distribution control
Matthew J. Holland
In this work, we consider the notion of "criterion collapse," in which
optimization of one metric implies optimality in another, with a particular
focus on conditions for collapse into error probability minimizers under a wide
variety of learning criteria, ranging from DRO and OCE risks (CVaR, tilted ERM)
to non-monotonic criteria underlying recent ascent-descent algorithms explored
in the literature (Flooding, SoftAD). We show how collapse in the context of
losses with a Bernoulli distribution goes far beyond existing results for CVaR
and DRO, then expand our scope to include surrogate losses, showing conditions
where monotonic criteria such as tilted ERM cannot avoid collapse, whereas
non-monotonic alternatives can.