Exploring the Sharpened Cosine Similarity
Skyler Wu Fred Lu Edward Raff James Holt
Convolutional layers have long served as the primary workhorse for image
classification. Recently, an alternative to convolution was proposed using the
Sharpened Cosine Similarity (SCS), which in theory may serve as a better
feature detector. While multiple sources report promising results, there has
not been to date a full-scale empirical analysis of neural network performance
using these new layers. In our work, we explore SCS's parameter behavior and
potential as a drop-in replacement for convolutions in multiple CNN
architectures benchmarked on CIFAR-10. We find that while SCS may not yield
significant increases in accuracy, it may learn more interpretable
representations. We also find that, in some circumstances, SCS may confer a
slight increase in adversarial robustness.