VAULT: Vigilant Adversarial Updates via LLM-Driven Retrieval-Augmented Generation for NLI
This work addresses robustness issues in NLI models for AI applications, offering an incremental improvement through automated adversarial data curation.
The paper tackled the problem of improving robustness in natural language inference (NLI) models by introducing VAULT, an automated adversarial retrieval-augmented generation pipeline, which increased RoBERTa-base accuracy by up to 17.32% on benchmarks like MultiNLI.
We introduce VAULT, a fully automated adversarial RAG pipeline that systematically uncovers and remedies weaknesses in NLI models through three stages: retrieval, adversarial generation, and iterative retraining. First, we perform balanced few-shot retrieval by embedding premises with both semantic (BGE) and lexical (BM25) similarity. Next, we assemble these contexts into LLM prompts to generate adversarial hypotheses, which are then validated by an LLM ensemble for label fidelity. Finally, the validated adversarial examples are injected back into the training set at increasing mixing ratios, progressively fortifying a zero-shot RoBERTa-base model.On standard benchmarks, VAULT elevates RoBERTa-base accuracy from 88.48% to 92.60% on SNLI +4.12%, from 75.04% to 80.95% on ANLI +5.91%, and from 54.67% to 71.99% on MultiNLI +17.32%. It also consistently outperforms prior in-context adversarial methods by up to 2.0% across datasets. By automating high-quality adversarial data curation at scale, VAULT enables rapid, human-independent robustness improvements in NLI inference tasks.