Sparse Lifting of Dense Vectors: Unifying Word and Sentence Representations
This addresses memory and computational challenges in natural language processing applications, offering a more interpretable and efficient alternative to existing dense vector methods.
The paper tackles the problem of uninterpretable and computationally expensive dense word and sentence vectors by proposing a novel representation model that projects dense vectors into a higher-dimensional space to achieve highly sparse and binary representations, while preserving pairwise inner products, resulting in consistent improvement in empirical evaluations.
As the first step in automated natural language processing, representing words and sentences is of central importance and has attracted significant research attention. Different approaches, from the early one-hot and bag-of-words representation to more recent distributional dense and sparse representations, were proposed. Despite the successful results that have been achieved, such vectors tend to consist of uninterpretable components and face nontrivial challenge in both memory and computational requirement in practical applications. In this paper, we designed a novel representation model that projects dense word vectors into a higher dimensional space and favors a highly sparse and binary representation of word vectors with potentially interpretable components, while trying to maintain pairwise inner products between original vectors as much as possible. Computationally, our model is relaxed as a symmetric non-negative matrix factorization problem which admits a fast yet effective solution. In a series of empirical evaluations, the proposed model exhibited consistent improvement and high potential in practical applications.