Word2Vec is a special case of Kernel Correspondence Analysis and Kernels for Natural Language Processing
This work addresses a memory bottleneck in NLP for researchers, offering an incremental improvement over existing methods.
The paper tackles the problem of applying correspondence analysis (CA) to natural language processing by introducing a memory-efficient method using delayed evaluation in randomized singular value decomposition, and proposes a tail-cut kernel that outperforms existing word-vector representation methods.
We show that correspondence analysis (CA) is equivalent to defining a Gini index with appropriately scaled one-hot encoding. Using this relation, we introduce a nonlinear kernel extension to CA. This extended CA gives a known analysis for natural language via specialized kernels that use an appropriate contingency table. We propose a semi-supervised CA, which is a special case of the kernel extension to CA. Because CA requires excessive memory if applied to numerous categories, CA has not been used for natural language processing. We address this problem by introducing delayed evaluation to randomized singular value decomposition. The memory-efficient CA is then applied to a word-vector representation task. We propose a tail-cut kernel, which is an extension to the skip-gram within the kernel extension to CA. Our tail-cut kernel outperforms existing word-vector representation methods.