Compressing Word Embeddings
This addresses memory efficiency for NLP applications, but is incremental as it builds on existing embedding methods.
The paper tackles the problem of high memory usage in word embeddings by proposing a compression scheme that reduces the memory footprint by a factor of 10 with minimal performance impact, and explores a binary factorization for better interpretability.
Recent methods for learning vector space representations of words have succeeded in capturing fine-grained semantic and syntactic regularities using vector arithmetic. However, these vector space representations (created through large-scale text analysis) are typically stored verbatim, since their internal structure is opaque. Using word-analogy tests to monitor the level of detail stored in compressed re-representations of the same vector space, the trade-offs between the reduction in memory usage and expressiveness are investigated. A simple scheme is outlined that can reduce the memory footprint of a state-of-the-art embedding by a factor of 10, with only minimal impact on performance. Then, using the same `bit budget', a binary (approximate) factorisation of the same space is also explored, with the aim of creating an equivalent representation with better interpretability.