R-Block: Regularized Block of Dropout for convolutional networks
Liqi Wang Qiya Hu
Dropout as a regularization technique is widely used in fully connected
layers while is less effective in convolutional layers. Therefore more
structured forms of dropout have been proposed to regularize convolutional
networks. The disadvantage of these methods is that the randomness introduced
causes inconsistency between training and inference. In this paper, we apply a
mutual learning training strategy for convolutional layer regularization,
namely R-Block, which forces two outputs of the generated difference maximizing
sub models to be consistent with each other. Concretely, R-Block minimizes the
losses between the output distributions of two sub models with different drop
regions for each sample in the training dataset. We design two approaches to
construct such sub models. Our experiments demonstrate that R-Block achieves
better performance than other existing structured dropout variants. We also
demonstrate that our approaches to construct sub models outperforms others.