Towards Scalable and Reliable Capsule Networks for Challenging NLP Applications
This work addresses scalability and reliability problems for researchers and practitioners using capsule networks in challenging NLP applications, though it appears incremental.
The paper tackled scalability and reliability issues in capsule networks for NLP by introducing an agreement score, adaptive optimizer, and compression techniques, achieving considerable improvements over strong competitors on multi-label text classification and question answering tasks, with best results in low-resource settings.
Obstacles hindering the development of capsule networks for challenging NLP applications include poor scalability to large output spaces and less reliable routing processes. In this paper, we introduce: 1) an agreement score to evaluate the performance of routing processes at instance level; 2) an adaptive optimizer to enhance the reliability of routing; 3) capsule compression and partial routing to improve the scalability of capsule networks. We validate our approach on two NLP tasks, namely: multi-label text classification and question answering. Experimental results show that our approach considerably improves over strong competitors on both tasks. In addition, we gain the best results in low-resource settings with few training instances.