BSBench: will your LLM find the largest prime number?
This addresses the issue of evaluating LLM robustness for researchers and developers, though it appears incremental as it builds on existing benchmarking approaches.
The authors tackled the problem of benchmarking LLMs on questions with no reasonable answers, proposing a benchmark and method to modify datasets, and found that existing models perform far from perfect on such tasks.
We propose that benchmarking LLMs on questions which have no reasonable answer actually isn't as silly as it sounds. We also present a benchmark that allows such testing and a method to modify the existing datasets, and discover that existing models demonstrate a performance far from the perfect on such questions. Our code and data artifacts are available at https://github.com/L3G5/impossible-bench