Learning in Convolutional Neural Networks Accelerated by Transfer
  Entropy
Adrian Moldovan Angel Caţaron Răzvan Andonie
Recently, there is a growing interest in applying Transfer Entropy (TE) in
quantifying the effective connectivity between artificial neurons. In a
feedforward network, the TE can be used to quantify the relationships between
neuron output pairs located in different layers. Our focus is on how to include
the TE in the learning mechanisms of a Convolutional Neural Network (CNN)
architecture. We introduce a novel training mechanism for CNN architectures
which integrates the TE feedback connections. Adding the TE feedback parameter
accelerates the training process, as fewer epochs are needed. On the flip side,
it adds computational overhead to each epoch. According to our experiments on
CNN classifiers, to achieve a reasonable computational overhead--accuracy
trade-off, it is efficient to consider only the inter-neural information
transfer of a random subset of the neuron pairs from the last two fully
connected layers. The TE acts as a smoothing factor, generating stability and
becoming active only periodically, not after processing each input sample.
Therefore, we can consider the TE is in our model a slowly changing
meta-parameter.