Text-as-Signal: Quantitative Semantic Scoring with Embeddings, Logprobs, and Noise Reduction
This provides a configurable workflow for AI engineering tasks like corpus inspection and monitoring, but it is incremental as it builds on existing embedding and dimensionality reduction techniques.
The paper tackles the problem of converting text corpora into quantitative semantic signals by developing a pipeline that uses embeddings, logprob-based scoring, and noise reduction to analyze documents, applied to 11,922 Portuguese news articles about AI to enable semantic positioning and corpus characterization.
This paper presents a practical pipeline for turning text corpora into quantitative semantic signals. Each news item is represented as a full-document embedding, scored through logprob-based evaluation over a configurable positional dictionary, and projected onto a noise-reduced low-dimensional manifold for structural interpretation. In the present case study, the dictionary is instantiated as six semantic dimensions and applied to a corpus of 11,922 Portuguese news articles about Artificial Intelligence. The resulting identity space supports both document-level semantic positioning and corpus-level characterization through aggregated profiles. We show how Qwen embeddings, UMAP, semantic indicators derived directly from the model output space, and a three-stage anomaly-detection procedure combine into an operational text-as-signal workflow for AI engineering tasks such as corpus inspection, monitoring, and downstream analytical support. Because the identity layer is configurable, the same framework can be adapted to the requirements of different analytical streams rather than fixed to a universal schema.