Learning Neural Networks on SVD Boosted Latent Spaces for Semantic Classification
This work offers a method to reduce the computational burden for researchers and practitioners working with deep learning models on large vocabulary text classification tasks, potentially improving efficiency without sacrificing accuracy.
This paper addresses the challenge of high-dimensional input spaces in text classification by proposing the use of Singular Value Decomposition (SVD) to project the input into a lower-dimensional latent space. Neural networks trained on this SVD-boosted latent space achieve comparable or superior performance to those trained on the original high-dimensional space, while significantly reducing computational complexity.
The availability of large amounts of data and compelling computation power have made deep learning models much popular for text classification and sentiment analysis. Deep neural networks have achieved competitive performance on the above tasks when trained on naive text representations such as word count, term frequency, and binary matrix embeddings. However, many of the above representations result in the input space having a dimension of the order of the vocabulary size, which is enormous. This leads to a blow-up in the number of parameters to be learned, and the computational cost becomes infeasible when scaling to domains that require retaining a colossal vocabulary. This work proposes using singular value decomposition to transform the high dimensional input space to a lower-dimensional latent space. We show that neural networks trained on this lower-dimensional space are not only able to retain performance while savoring significant reduction in the computational complexity but, in many situations, also outperforms the classical neural networks trained on the native input space.