Natural Language Processing with Small Feed-Forward Networks
This work addresses resource constraints in environments like mobile phones, offering a practical solution for deploying NLP models efficiently.
The paper tackled the problem of high computational and memory costs in NLP models by demonstrating that small, shallow feed-forward networks can achieve near state-of-the-art results on various language tasks, with significant reductions in resource requirements.
We show that small and shallow feed-forward neural networks can achieve near state-of-the-art results on a range of unstructured and structured language processing tasks while being considerably cheaper in memory and computational requirements than deep recurrent models. Motivated by resource-constrained environments like mobile phones, we showcase simple techniques for obtaining such small neural network models, and investigate different tradeoffs when deciding how to allocate a small memory budget.