Transformation of Dense and Sparse Text Representations
This addresses a bottleneck in NLP by allowing researchers to leverage sparsity for explanation while maintaining compatibility with dense methods, though it appears incremental as it builds on existing transformation concepts.
The paper tackles the gap between dense and sparse text representations by proposing a Semantic Transformation method that bridges both spaces, enabling operations in sparse space and conversion back to dense representations, with experiments showing effectiveness in classification and natural language inference tasks.
Sparsity is regarded as a desirable property of representations, especially in terms of explanation. However, its usage has been limited due to the gap with dense representations. Most NLP research progresses in recent years are based on dense representations. Thus the desirable property of sparsity cannot be leveraged. Inspired by Fourier Transformation, in this paper, we propose a novel Semantic Transformation method to bridge the dense and sparse spaces, which can facilitate the NLP research to shift from dense space to sparse space or to jointly use both spaces. The key idea of the proposed approach is to use a Forward Transformation to transform dense representations to sparse representations. Then some useful operations in the sparse space can be performed over the sparse representations, and the sparse representations can be used directly to perform downstream tasks such as text classification and natural language inference. Then, a Backward Transformation can also be carried out to transform those processed sparse representations to dense representations. Experiments using classification tasks and natural language inference task show that the proposed Semantic Transformation is effective.