Idea First, Code Later: Disentangling Problem Solving from Code Generation in Evaluating LLMs for Competitive Programming
This work addresses the problem of accurately assessing LLMs' problem-solving abilities in competitive programming for researchers and developers, though it is incremental by refining evaluation methods rather than introducing a new paradigm.
The paper tackles the issue of conflating algorithmic reasoning with code implementation in evaluating LLMs for competitive programming by proposing to use natural-language editorials for both solution generation and evaluation. It finds that generating editorials before code improves solve rates for some LLMs, with larger gains using gold editorials, but models still struggle with implementation, revealing a persistent problem-solving bottleneck.
Large Language Models (LLMs) increasingly succeed on competitive programming problems, yet existing evaluations conflate algorithmic reasoning with code-level implementation. We argue that competitive programming is fundamentally a problem-solving task and propose centering natural-language editorials in both solution generation and evaluation. Generating an editorial prior to code improves solve rates for some LLMs, with substantially larger gains when using expertly written gold editorials. However, even with gold editorials, models continue to struggle with implementation, while the gap between generated and gold editorials reveals a persistent problem-solving bottleneck in specifying correct and complete algorithms. Beyond pass/fail metrics, we diagnose reasoning errors by comparing model-generated editorials to gold standards using expert annotations and validate an LLM-as-a-judge protocol for scalable evaluation. We introduce a dataset of 83 ICPC-style problems with gold editorials and full test suites, and evaluate 19 LLMs, arguing that future benchmarks should explicitly separate problem solving from implementation.