On Self-improving Token Embeddings
This provides a method for improving token embeddings in topically homogeneous domains, such as disaster narratives, but it is incremental as it builds on existing embedding techniques.
The paper tackles the problem of refining pre-trained token embeddings for domain-specific corpora by updating representations using neighboring tokens, which addresses out-of-vocabulary issues and results in more meaningful embeddings compared to general-purpose vectors, as demonstrated with storm event narratives from the NOAA database.
This article introduces a novel and fast method for refining pre-trained static word or, more generally, token embeddings. By incorporating the embeddings of neighboring tokens in text corpora, it continuously updates the representation of each token, including those without pre-assigned embeddings. This approach effectively addresses the out-of-vocabulary problem, too. Operating independently of large language models and shallow neural networks, it enables versatile applications such as corpus exploration, conceptual search, and word sense disambiguation. The method is designed to enhance token representations within topically homogeneous corpora, where the vocabulary is restricted to a specific domain, resulting in more meaningful embeddings compared to general-purpose pre-trained vectors. As an example, the methodology is applied to explore storm events and their impacts on infrastructure and communities using narratives from a subset of the NOAA Storm Events database. The article also demonstrates how the approach improves the representation of storm-related terms over time, providing valuable insights into the evolving nature of disaster narratives.