Cross Entropy in Deep Learning of Classifiers Is Unnecessary -- ISBE
  Error is All You Need
Wladyslaw Skarbek
In deep learning classifiers, the cost function usually takes the form of a
combination of SoftMax and CrossEntropy functions. The SoftMax unit transforms
the scores predicted by the model network into assessments of the degree
(probabilities) of an object's membership to a given class. On the other hand,
CrossEntropy measures the divergence of this prediction from the distribution
of target scores. This work introduces the ISBE functionality, justifying the
thesis about the redundancy of cross entropy computation in deep learning of
classifiers. Not only can we omit the calculation of entropy, but also, during
back-propagation, there is no need to direct the error to the normalization
unit for its backward transformation. Instead, the error is sent directly to
the model's network. Using examples of perceptron and convolutional networks as
classifiers of images from the MNIST collection, it is observed for ISBE that
results are not degraded with SoftMax only, but also with other activation
functions such as Sigmoid, Tanh, or their hard variants HardSigmoid and
HardTanh. Moreover, up to three percent of time is saved within the total time
of forward and backward stages. The article is addressed mainly to programmers
and students interested in deep model learning. For example, it illustrates in
code snippets possible ways to implement ISBE units, but also formally proves
that the softmax trick only applies to the class of softmax functions with
relocations.