A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios
This work addresses the need for better robustness assessment in AI systems for researchers and practitioners, but it is incremental as it focuses on a specific scenario without broad SOTA claims.
The paper tackles the problem of measuring the robustness of large language models in non-adversarial scenarios by proposing a novel metric and evaluating models on datasets with naturally-occurring perturbations, showing benefits through empirical evaluation.
We evaluate the robustness of several large language models on multiple datasets. Robustness here refers to the relative insensitivity of the model's answers to meaning-preserving variants of their input. Benchmark datasets are constructed by introducing naturally-occurring, non-malicious perturbations, or by generating semantically equivalent paraphrases of input questions or statements. We further propose a novel metric for assessing a model robustness, and demonstrate its benefits in the non-adversarial scenario by empirical evaluation of several models on the created datasets.