On Learning and Learned Data Representation by Capsule Networks
This work provides incremental insights into CapsNet mechanisms for researchers in neural network representation learning.
The paper investigates how routing affects CapsNet model fitting, how capsule representations help discover global data structures, and how learned representations adapt to new tasks, finding that routing determines information certainty related to fitness, capsules enable more meaningful 2D embeddings than CNNs in structured data, and capsules are less coupled and more adaptive than CNN neurons.
In this work, we investigate the following: 1) how the routing affects the CapsNet model fitting; 2) how the representation using capsules helps discover global structures in data distribution, and; 3) how the learned data representation adapts and generalizes to new tasks. Our investigation yielded the results some of which have been mentioned in the original paper of CapsNet, they are: 1) the routing operation determines the certainty with which a layer of capsules pass information to the layer above and the appropriate level of certainty is related to the model fitness; 2) in a designed experiment using data with a known 2D structure, capsule representations enable a more meaningful 2D manifold embedding than neurons do in a standard convolutional neural network (CNN), and; 3) compared with neurons of the standard CNN, capsules of successive layers are less coupled and more adaptive to new data distribution.