Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis
This work addresses the robustness problem for users of few-shot prompting techniques in LLMs, though it is incremental as it builds on existing prompt engineering methods.
The study investigated the robustness of large language models (LLMs) in multi-hop reasoning tasks by introducing domain-agnostic perturbations, finding that models are sensitive to certain perturbations like synonym replacements, and demonstrated that increasing the proportion of perturbed exemplars in prompts improves robustness.
Recent advances in prompt engineering enable large language models (LLMs) to solve multi-hop logical reasoning problems with impressive accuracy. However, there is little existing work investigating the robustness of LLMs with few-shot prompting techniques. Therefore, we introduce a systematic approach to test the robustness of LLMs in multi-hop reasoning tasks via domain-agnostic perturbations. We include perturbations at multiple levels of abstractions (e.g. lexical perturbations such as typos, and semantic perturbations such as the inclusion of intermediate reasoning steps in the questions) to conduct behavioral analysis on the LLMs. Throughout our experiments, we find that models are more sensitive to certain perturbations such as replacing words with their synonyms. We also demonstrate that increasing the proportion of perturbed exemplars in the prompts improves the robustness of few-shot prompting methods.