Exploring the Sharpened Cosine Similarity
This work addresses the need for alternative feature detectors in neural networks, though it appears incremental as SCS shows limited practical advantages over existing methods.
The researchers investigated whether Sharpened Cosine Similarity (SCS) layers could effectively replace convolutional layers in CNNs for image classification on CIFAR-10, finding they did not significantly improve accuracy but might offer more interpretable representations and slight adversarial robustness gains.
Convolutional layers have long served as the primary workhorse for image classification. Recently, an alternative to convolution was proposed using the Sharpened Cosine Similarity (SCS), which in theory may serve as a better feature detector. While multiple sources report promising results, there has not been to date a full-scale empirical analysis of neural network performance using these new layers. In our work, we explore SCS's parameter behavior and potential as a drop-in replacement for convolutions in multiple CNN architectures benchmarked on CIFAR-10. We find that while SCS may not yield significant increases in accuracy, it may learn more interpretable representations. We also find that, in some circumstances, SCS may confer a slight increase in adversarial robustness.