A Hybrid of Generative and Discriminative Models Based on the
  Gaussian-coupled Softmax Layer
Hideaki Hayashi
Generative models have advantageous characteristics for classification tasks
such as the availability of unsupervised data and calibrated confidence,
whereas discriminative models have advantages in terms of the simplicity of
their model structures and learning algorithms and their ability to outperform
their generative counterparts. In this paper, we propose a method to train a
hybrid of discriminative and generative models in a single neural network (NN),
which exhibits the characteristics of both models. The key idea is the
Gaussian-coupled softmax layer, which is a fully connected layer with a softmax
activation function coupled with Gaussian distributions. This layer can be
embedded into an NN-based classifier and allows the classifier to estimate both
the class posterior distribution and the class-conditional data distribution.
We demonstrate that the proposed hybrid model can be applied to semi-supervised
learning and confidence calibration.