Adversarial Semantic Collisions
This work addresses a security and reliability issue for users of NLP systems in applications such as retrieval and summarization, though it is incremental as it builds on existing adversarial attack methods.
The paper tackles the problem of semantic collisions, where texts unrelated in meaning are incorrectly judged as similar by NLP models, by developing gradient-based methods to generate such collisions and demonstrating vulnerabilities in state-of-the-art models for tasks like paraphrase identification and document retrieval, with results showing that crafted collisions can shift a document's retrieval rank from 1000 to top 3.
We study semantic collisions: texts that are semantically unrelated but judged as similar by NLP models. We develop gradient-based approaches for generating semantic collisions and demonstrate that state-of-the-art models for many tasks which rely on analyzing the meaning and similarity of texts-- including paraphrase identification, document retrieval, response suggestion, and extractive summarization-- are vulnerable to semantic collisions. For example, given a target query, inserting a crafted collision into an irrelevant document can shift its retrieval rank from 1000 to top 3. We show how to generate semantic collisions that evade perplexity-based filtering and discuss other potential mitigations. Our code is available at https://github.com/csong27/collision-bert.