Linearly Constrained Weights: Reducing Activation Shift for Faster
  Training of Neural Networks
Takuro Kutsuna
In this paper, we first identify activation shift, a simple but remarkable
phenomenon in a neural network in which the preactivation value of a neuron has
non-zero mean that depends on the angle between the weight vector of the neuron
and the mean of the activation vector in the previous layer. We then propose
linearly constrained weights (LCW) to reduce the activation shift in both fully
connected and convolutional layers. The impact of reducing the activation shift
in a neural network is studied from the perspective of how the variance of
variables in the network changes through layer operations in both forward and
backward chains. We also discuss its relationship to the vanishing gradient
problem. Experimental results show that LCW enables a deep feedforward network
with sigmoid activation functions to be trained efficiently by resolving the
vanishing gradient problem. Moreover, combined with batch normalization, LCW
improves generalization performance of both feedforward and convolutional
networks.