Classification with Deep Neural Networks and Logistic Loss
Zihan Zhang Lei Shi Ding-Xuan Zhou
Deep neural networks (DNNs) trained with the logistic loss (i.e., the cross
entropy loss) have made impressive advancements in various binary
classification tasks. However, generalization analysis for binary
classification with DNNs and logistic loss remains scarce. The unboundedness of
the target function for the logistic loss is the main obstacle to deriving
satisfying generalization bounds. In this paper, we aim to fill this gap by
establishing a novel and elegant oracle-type inequality, which enables us to
deal with the boundedness restriction of the target function, and using it to
derive sharp convergence rates for fully connected ReLU DNN classifiers trained
with logistic loss. In particular, we obtain optimal convergence rates (up to
log factors) only requiring the H\"older smoothness of the conditional class
probability $\eta$ of data. Moreover, we consider a compositional assumption
that requires $\eta$ to be the composition of several vector-valued functions
of which each component function is either a maximum value function or a
H\"older smooth function only depending on a small number of its input
variables. Under this assumption, we derive optimal convergence rates (up to
log factors) which are independent of the input dimension of data. This result
explains why DNN classifiers can perform well in practical high-dimensional
classification problems. Besides the novel oracle-type inequality, the sharp
convergence rates given in our paper also owe to a tight error bound for
approximating the natural logarithm function near zero (where it is unbounded)
by ReLU DNNs. In addition, we justify our claims for the optimality of rates by
proving corresponding minimax lower bounds. All these results are new in the
literature and will deepen our theoretical understanding of classification with
DNNs.