RUPBench: Benchmarking Reasoning Under Perturbations for Robustness Evaluation in Large Language Models
This work addresses the need for reliable LLM performance in real-world environments by providing a systematic benchmark for robustness evaluation, though it is incremental as it builds on existing benchmarking efforts.
The authors tackled the problem of evaluating the robustness of large language models (LLMs) to adversarial inputs by introducing RUPBench, a benchmark with 15 reasoning datasets and nine textual perturbations, finding that larger models tend to be more robust to these perturbations.
With the increasing use of large language models (LLMs), ensuring reliable performance in diverse, real-world environments is essential. Despite their remarkable achievements, LLMs often struggle with adversarial inputs, significantly impacting their effectiveness in practical applications. To systematically understand the robustness of LLMs, we present RUPBench, a comprehensive benchmark designed to evaluate LLM robustness across diverse reasoning tasks. Our benchmark incorporates 15 reasoning datasets, categorized into commonsense, arithmetic, logical, and knowledge-intensive reasoning, and introduces nine types of textual perturbations at lexical, syntactic, and semantic levels. By examining the performance of state-of-the-art LLMs such as GPT-4o, Llama3, Phi-3, and Gemma on both original and perturbed datasets, we provide a detailed analysis of their robustness and error patterns. Our findings highlight that larger models tend to exhibit greater robustness to perturbations. Additionally, common error types are identified through manual inspection, revealing specific challenges faced by LLMs in different reasoning contexts. This work provides insights into areas where LLMs need further improvement to handle diverse and noisy inputs effectively.