Mixture of Expert/Imitator Networks: Scalable Semi-supervised Learning Framework
This addresses the issue of limited labeled data for researchers and practitioners in natural language processing, though it appears incremental as it builds on existing semi-supervised learning approaches.
The paper tackles the problem of labeled data scarcity in text classification by proposing a scalable semi-supervised learning method called Mixture of Expert/Imitator Networks, which improves performance over baseline DNNs and shows better results with more unlabeled data.
The current success of deep neural networks (DNNs) in an increasingly broad range of tasks involving artificial intelligence strongly depends on the quality and quantity of labeled training data. In general, the scarcity of labeled data, which is often observed in many natural language processing tasks, is one of the most important issues to be addressed. Semi-supervised learning (SSL) is a promising approach to overcoming this issue by incorporating a large amount of unlabeled data. In this paper, we propose a novel scalable method of SSL for text classification tasks. The unique property of our method, Mixture of Expert/Imitator Networks, is that imitator networks learn to "imitate" the estimated label distribution of the expert network over the unlabeled data, which potentially contributes a set of features for the classification. Our experiments demonstrate that the proposed method consistently improves the performance of several types of baseline DNNs. We also demonstrate that our method has the more data, better performance property with promising scalability to the amount of unlabeled data.