Mechanism of feature learning in convolutional neural networks
Daniel Beaglehole Adityanarayanan Radhakrishnan Parthe Pandit Mikhail Belkin
Understanding the mechanism of how convolutional neural networks learn
features from image data is a fundamental problem in machine learning and
computer vision. In this work, we identify such a mechanism. We posit the
Convolutional Neural Feature Ansatz, which states that covariances of filters
in any convolutional layer are proportional to the average gradient outer
product (AGOP) taken with respect to patches of the input to that layer. We
present extensive empirical evidence for our ansatz, including identifying high
correlation between covariances of filters and patch-based AGOPs for
convolutional layers in standard neural architectures, such as AlexNet, VGG,
and ResNets pre-trained on ImageNet. We also provide supporting theoretical
evidence. We then demonstrate the generality of our result by using the
patch-based AGOP to enable deep feature learning in convolutional kernel
machines. We refer to the resulting algorithm as (Deep) ConvRFM and show that
our algorithm recovers similar features to deep convolutional networks
including the notable emergence of edge detectors. Moreover, we find that Deep
ConvRFM overcomes previously identified limitations of convolutional kernels,
such as their inability to adapt to local signals in images and, as a result,
leads to sizable performance improvement over fixed convolutional kernels.