ThrowBench: Benchmarking LLMs by Predicting Runtime Exceptions
This work addresses the need for broader benchmarking to identify weak points in code LLMs for researchers and developers, though it is incremental as it extends existing evaluation methods.
The authors tackled the problem of assessing code understanding in LLMs beyond code synthesis by introducing ThrowBench, a benchmark of 2,400 short programs in four languages that test prediction of runtime exceptions, and found modest performance with F1 scores ranging from 19% to 38% across six state-of-the-art models.
Modern Large Language Models (LLMs) have shown astounding capabilities of code understanding and synthesis. In order to assess such capabilities, several benchmarks have been devised (e.g., HumanEval). However, most benchmarks focus on code synthesis from natural language instructions. Hence, such benchmarks do not test for other forms of code understanding. Moreover, there have been concerns about contamination and leakage. That is, benchmark problems (or closely related problems) may appear in training set, strongly biasing benchmark results. In this work we investigate whether large language models can correctly predict runtime program behavior. To this end, we introduce ThrowBench, a benchmark consisting of over 2,400 short user-written programs written in four different programming languages. The majority of these programs throw an exception during runtime (due to a bug). LLMs are asked to predict whether a presented program throws an exception and, if so, which one. Evaluating our benchmark on six state-of-the-art code LLMs we see modest performance ranging from 19 to 38% (F1 score). Benchmarking a wider set of code capabilities could improve the assessment of code LLMs and help identify weak points in current models. Moreover, as ground-truth answers have been determined through program execution, leakage is not a concern. We release ThrowBench as well as all of our results together with this work.