Squeezing bottlenecks: exploring the limits of autoencoder semantic representation capabilities
This work addresses the challenge of optimizing autoencoder architectures for text representation, which is incremental as it builds on existing methods with specific improvements.
The study tackled the problem of modeling text data with autoencoders by exploring model suitability, proposing new reconstruction metrics, and developing an automatic method to find critical bottleneck dimensionality, resulting in the identification of limits below which structural information is lost.
We present a comprehensive study on the use of autoencoders for modelling text data, in which (differently from previous studies) we focus our attention on the following issues: i) we explore the suitability of two different models bDA and rsDA for constructing deep autoencoders for text data at the sentence level; ii) we propose and evaluate two novel metrics for better assessing the text-reconstruction capabilities of autoencoders; and iii) we propose an automatic method to find the critical bottleneck dimensionality for text language representations (below which structural information is lost).