Autoencoder-Based Framework to Capture Vocabulary Quality in NLP
This work addresses the need for better vocabulary evaluation in NLP to improve dataset curation and model design, though it appears incremental by enhancing existing metrics rather than introducing a new paradigm.
The paper tackles the problem of inadequate traditional metrics for capturing vocabulary quality in NLP by introducing an autoencoder-based framework that uses neural network capacity as a proxy for richness, diversity, and complexity, validated on datasets like DIFrauD and Project Gutenberg to offer practical guidance for dataset curation and model design.
Linguistic richness is essential for advancing natural language processing (NLP), as dataset characteristics often directly influence model performance. However, traditional metrics such as Type-Token Ratio (TTR), Vocabulary Diversity (VOCD), and Measure of Lexical Text Diversity (MTLD) do not adequately capture contextual relationships, semantic richness, and structural complexity. In this paper, we introduce an autoencoder-based framework that uses neural network capacity as a proxy for vocabulary richness, diversity, and complexity, enabling a dynamic assessment of the interplay between vocabulary size, sentence structure, and contextual depth. We validate our approach on two distinct datasets: the DIFrauD dataset, which spans multiple domains of deceptive and fraudulent text, and the Project Gutenberg dataset, representing diverse languages, genres, and historical periods. Experimental results highlight the robustness and adaptability of our method, offering practical guidance for dataset curation and NLP model design. By enhancing traditional vocabulary evaluation, our work fosters the development of more context-aware, linguistically adaptive NLP systems.