An Experimental Study of Real-Life LLM-Proposed Performance Improvements
This work addresses the practical utility of LLMs for code optimization, showing incremental improvements but highlighting limitations compared to human expertise.
The study investigated whether LLMs can generate fast code for real-world Java tasks, finding that LLM-generated patches improved performance over baselines in most cases but were statistically outperformed by human solutions, with original LLM ideas rarely yielding substantial gains.
Large Language Models (LLMs) can generate code, but can they generate fast code? In this paper, we study this question using a dataset of 65 real-world tasks mined from open-source Java programs. We specifically select tasks where developers achieved significant speedups, and employ an automated pipeline to generate patches for these issues using two leading LLMs under four prompt variations. By rigorously benchmarking the results against the baseline and human-authored solutions, we demonstrate that LLM-generated code indeed improves performance over the baseline in most cases. However, patches proposed by human developers outperform LLM fixes by a statistically significant margin, indicating that LLMs often fall short of finding truly optimal solutions. We further find that LLM solutions are semantically identical or similar to the developer optimization idea in approximately two-thirds of cases, whereas they propose a more original idea in the remaining one-third. However, these original ideas only occasionally yield substantial performance gains.