Smaller Text Classifiers with Discriminative Cluster Embeddings
This addresses model size reduction for text classification, but it is incremental as it builds on existing embedding and clustering methods.
The paper tackled the problem of large model sizes in neural text classifiers by learning a hard word clustering end-to-end, reducing deployed model sizes while maintaining accuracy through parameter-efficient variations.
Word embedding parameters often dominate overall model sizes in neural methods for natural language processing. We reduce deployed model sizes of text classifiers by learning a hard word clustering in an end-to-end manner. We use the Gumbel-Softmax distribution to maximize over the latent clustering while minimizing the task loss. We propose variations that selectively assign additional parameters to words, which further improves accuracy while still remaining parameter-efficient.