Entropy-based Guidance of Deep Neural Networks for Accelerated
  Convergence and Improved Performance
Mackenzie J. Meni Ryan T. White Michael Mayo Kevin Pilkiewicz
Neural networks have dramatically increased our capacity to learn from large,
high-dimensional datasets across innumerable disciplines. However, their
decisions are not easily interpretable, their computational costs are high, and
building and training them are uncertain processes. To add structure to these
efforts, we derive new mathematical results to efficiently measure the changes
in entropy as fully-connected and convolutional neural networks process data,
and introduce entropy-based loss terms. Experiments in image compression and
image classification on benchmark datasets demonstrate these losses guide
neural networks to learn rich latent data representations in fewer dimensions,
converge in fewer training epochs, and achieve better test metrics.