Regularization and Optimal Multiclass Learning
Julian Asilis Siddartha Devic Shaddin Dughmi Vatsal Sharan Shang-Hua Teng
The quintessential learning algorithm of empirical risk minimization (ERM) is
known to fail in various settings for which uniform convergence does not
characterize learning. It is therefore unsurprising that the practice of
machine learning is rife with considerably richer algorithmic techniques for
successfully controlling model capacity. Nevertheless, no such technique or
principle has broken away from the pack to characterize optimal learning in
these more general settings.
  The purpose of this work is to characterize the role of regularization in
perhaps the simplest setting for which ERM fails: multiclass learning with
arbitrary label sets. Using one-inclusion graphs (OIGs), we exhibit optimal
learning algorithms that dovetail with tried-and-true algorithmic principles:
Occam's Razor as embodied by structural risk minimization (SRM), the principle
of maximum entropy, and Bayesian reasoning. Most notably, we introduce an
optimal learner which relaxes structural risk minimization on two dimensions:
it allows the regularization function to be "local" to datapoints, and uses an
unsupervised learning stage to learn this regularizer at the outset. We justify
these relaxations by showing that they are necessary: removing either dimension
fails to yield a near-optimal learner. We also extract from OIGs a
combinatorial sequence we term the Hall complexity, which is the first to
characterize a problem's transductive error rate exactly.
  Lastly, we introduce a generalization of OIGs and the transductive learning
setting to the agnostic case, where we show that optimal orientations of
Hamming graphs -- judged using nodes' outdegrees minus a system of
node-dependent credits -- characterize optimal learners exactly. We demonstrate
that an agnostic version of the Hall complexity again characterizes error rates
exactly, and exhibit an optimal learner using maximum entropy programs.