Deciding How to Decide: Dynamic Routing in Artificial Neural Networks
This addresses the challenge of improving efficiency and specialization in neural networks for image classification, though it appears incremental as it compares existing routing strategies.
The paper tackles the problem of training dynamically-routed neural networks, where different inputs take different paths, and finds that these networks specialize layers and branches for distinct image categories and outperform static networks under fixed computational budgets.
We propose and systematically evaluate three strategies for training dynamically-routed artificial neural networks: graphs of learned transformations through which different input signals may take different paths. Though some approaches have advantages over others, the resulting networks are often qualitatively similar. We find that, in dynamically-routed networks trained to classify images, layers and branches become specialized to process distinct categories of images. Additionally, given a fixed computational budget, dynamically-routed networks tend to perform better than comparable statically-routed networks.