Factor Augmented Supervised Learning with Text Embeddings
This addresses computational inefficiency in text-based machine learning applications, but it is incremental as it builds on existing embedding and dimension reduction techniques.
The paper tackles the problem of high-dimensional text embeddings from large language models impeding efficiency in downstream tasks by proposing AEALT, a supervised factor-augmented framework that incorporates dimension reduction, resulting in substantial gains over vanilla embeddings and standard methods in classification, anomaly detection, and prediction tasks.
Large language models (LLMs) generate text embeddings from text data, producing vector representations that capture the semantic meaning and contextual relationships of words. However, the high dimensionality of these embeddings often impedes efficiency and drives up computational cost in downstream tasks. To address this, we propose AutoEncoder-Augmented Learning with Text (AEALT), a supervised, factor-augmented framework that incorporates dimension reduction directly into pre-trained LLM workflows. First, we extract embeddings from text documents; next, we pass them through a supervised augmented autoencoder to learn low-dimensional, task-relevant latent factors. By modeling the nonlinear structure of complex embeddings, AEALT outperforms conventional deep-learning approaches that rely on raw embeddings. We validate its broad applicability with extensive experiments on classification, anomaly detection, and prediction tasks using multiple real-world public datasets. Numerical results demonstrate that AEALT yields substantial gains over both vanilla embeddings and several standard dimension reduction methods.