On the Robustness of Text Vectorizers
This addresses robustness issues in NLP for practitioners using text vectorization, though it is incremental as it builds on existing theoretical frameworks.
The paper tackles the problem of robustness in text vectorizers to discrete input changes, such as word replacements, by formally proving that popular embedding schemes like TF-IDF and doc2vec exhibit robustness with quantitative bounds dependent on document length.
A fundamental issue in machine learning is the robustness of the model with respect to changes in the input. In natural language processing, models typically contain a first embedding layer, transforming a sequence of tokens into vector representations. While the robustness with respect to changes of continuous inputs is well-understood, the situation is less clear when considering discrete changes, for instance replacing a word by another in an input sentence. Our work formally proves that popular embedding schemes, such as concatenation, TF-IDF, and Paragraph Vector (a.k.a. doc2vec), exhibit robustness in the Hölder or Lipschitz sense with respect to the Hamming distance. We provide quantitative bounds for these schemes and demonstrate how the constants involved are affected by the length of the document. These findings are exemplified through a series of numerical examples.