RETVec: Resilient and Efficient Text Vectorizer
This addresses the need for robust text processing tools in NLP applications, though it appears incremental as it builds on existing vectorization methods with specific enhancements.
The paper tackles the problem of creating a text vectorizer that is efficient, resilient to typos and adversarial attacks, and multilingual, resulting in RETVec, which demonstrates competitive performance with significantly improved resilience in evaluations.
This paper describes RETVec, an efficient, resilient, and multilingual text vectorizer designed for neural-based text processing. RETVec combines a novel character encoding with an optional small embedding model to embed words into a 256-dimensional vector space. The RETVec embedding model is pre-trained using pair-wise metric learning to be robust against typos and character-level adversarial attacks. In this paper, we evaluate and compare RETVec to state-of-the-art vectorizers and word embeddings on popular model architectures and datasets. These comparisons demonstrate that RETVec leads to competitive, multilingual models that are significantly more resilient to typos and adversarial text attacks. RETVec is available under the Apache 2 license at https://github.com/google-research/retvec.