Comprehensive Evaluation of Large Language Models on Software Engineering Tasks: A Multi-Task Benchmark
This work provides a multi-task benchmark for evaluating LLMs in software engineering, addressing a gap for researchers and practitioners, though it is incremental as it builds on existing evaluation methods.
The paper tackled the lack of comprehensive benchmarks for large language models in software engineering by evaluating 11 models across five tasks, finding significant variations in efficiency and cost, with coding tasks achieving 100% success but research tasks being more challenging at 90.9%.
Large Language Models (LLMs) have demonstrated remarkable capabilities in software engineering, yet comprehensive benchmarks covering diverse SE activities remain limited. We present a multi-task evaluation of 11 state-of-the-art LLMs across five representative software engineering tasks: bug fixing, feature development, code refactoring, technical copywriting, and research synthesis. Our automated verification framework measures both output quality and completion efficiency. Key findings reveal that (1) models achieving identical perfect scores exhibit 22x variation in completion time, 49x variation in tool efficiency, and 53x variation in estimated cost; (2) tool usage frequency shows no correlation with success (r = 0.077, p = 0.575) - one model used 917 tool calls while another solved the same task with 3 calls; (3) we identify two distinct inefficiency patterns: loop inefficiency and inference inefficiency; and (4) coding tasks achieve 100 percent success while research tasks present greater challenges (90.9 percent). We release all experimental data, verification scripts, and analysis code for full reproducibility.