Limitation of capsule networks
This addresses a theoretical limitation in capsule networks for researchers in deep learning, but it is incremental as it builds on existing routing methods.
The paper proves that state-of-the-art routing procedures like EM-routing and routing-by-agreement limit capsule networks to symmetric functions, preventing them from distinguishing inputs and their negatives, which negatively affects training. It presents an incremental improvement to solve this limitation and stabilize training.
A recently proposed method in deep learning groups multiple neurons to capsules such that each capsule represents an object or part of an object. Routing algorithms route the output of capsules from lower-level layers to upper-level layers. In this paper, we prove that state-of-the-art routing procedures decrease the expressivity of capsule networks. More precisely, it is shown that EM-routing and routing-by-agreement prevent capsule networks from distinguishing inputs and their negative counterpart. Therefore, only symmetric functions can be expressed by capsule networks, and it can be concluded that they are not universal approximators. We also theoretically motivate and empirically show that this limitation affects the training of deep capsule networks negatively. Therefore, we present an incremental improvement for state-of-the-art routing algorithms that solves the aforementioned limitation and stabilizes the training of capsule networks.