Fundamental Limits of Deep Learning-Based Binary Classifiers Trained
  with Hinge Loss
Tilahun M. Getu Georges Kaddoum
Although deep learning (DL) has led to several breakthroughs in many
disciplines as diverse as chemistry, computer science, electrical engineering,
mathematics, medicine, neuroscience, and physics, a comprehensive understanding
of why and how DL is empirically successful remains fundamentally elusive. To
attack this fundamental problem and unravel the mysteries behind DL's empirical
successes, significant innovations toward a unified theory of DL have been
made. These innovations encompass nearly fundamental advances in optimization,
generalization, and approximation. Despite these advances, however, no work to
date has offered a way to quantify the testing performance of a DL-based
algorithm employed to solve a pattern classification problem. To overcome this
fundamental challenge in part, this paper exposes the fundamental testing
performance limits of DL-based binary classifiers trained with hinge loss. For
binary classifiers that are based on deep rectified linear unit (ReLU)
feedforward neural networks (FNNs) and ones that are based on deep FNNs with
ReLU and Tanh activation, we derive their respective novel asymptotic testing
performance limits. The derived testing performance limits are validated by
extensive computer experiments.