Capsule Networks with Max-Min Normalization
This addresses a bottleneck in Capsule Networks for researchers and practitioners, offering an incremental improvement in routing efficiency and accuracy.
The paper tackled the problem of suboptimal capsule couplings in Capsule Networks due to Softmax normalization, proposing Max-Min normalization instead, which improved test accuracy across five datasets and achieved a test error of 0.20% on MNIST with a single model and 0.17% with a 3-model majority vote.
Capsule Networks (CapsNet) use the Softmax function to convert the logits of the routing coefficients into a set of normalized values that signify the assignment probabilities between capsules in adjacent layers. We show that the use of Softmax prevents capsule layers from forming optimal couplings between lower and higher-level capsules. Softmax constrains the dynamic range of the routing coefficients and leads to probabilities that remain mostly uniform after several routing iterations. Instead, we propose the use of Max-Min normalization. Max-Min performs a scale-invariant normalization of the logits that allows each lower-level capsule to take on an independent value, constrained only by the bounds of normalization. Max-Min provides consistent improvement in test accuracy across five datasets and allows more routing iterations without a decrease in network performance. A single CapsNet trained using Max-Min achieves an improved test error of 0.20% on the MNIST dataset. With a simple 3-model majority vote, we achieve a test error of 0.17% on MNIST.