No Routing Needed Between Capsules
This work addresses the computational bottleneck in capsule networks for researchers and practitioners, offering a more efficient alternative with improved performance on structured datasets like MNIST, though it is incremental as it builds on existing capsule concepts.
The paper tackles the computational inefficiency of capsule networks by introducing Homogeneous Vector Capsules (HVCs) that use element-wise multiplication instead of matrix multiplication, eliminating the need for routing mechanisms. On MNIST, it achieves state-of-the-art accuracy of 99.87% with an ensemble and 99.83% with a single model, using 5.5x fewer parameters and 4x fewer training epochs compared to prior capsule networks.
Most capsule network designs rely on traditional matrix multiplication between capsule layers and computationally expensive routing mechanisms to deal with the capsule dimensional entanglement that the matrix multiplication introduces. By using Homogeneous Vector Capsules (HVCs), which use element-wise multiplication rather than matrix multiplication, the dimensions of the capsules remain unentangled. In this work, we study HVCs as applied to the highly structured MNIST dataset in order to produce a direct comparison to the capsule research direction of Geoffrey Hinton, et al. In our study, we show that a simple convolutional neural network using HVCs performs as well as the prior best performing capsule network on MNIST using 5.5x fewer parameters, 4x fewer training epochs, no reconstruction sub-network, and requiring no routing mechanism. The addition of multiple classification branches to the network establishes a new state of the art for the MNIST dataset with an accuracy of 99.87% for an ensemble of these models, as well as establishing a new state of the art for a single model (99.83% accurate).