CDR-Bench: Evaluating Faithful Execution of Compositional, Order-Sensitive Data Refinement Recipes
For researchers and practitioners using LLMs for data refinement, this benchmark reveals that current LLMs cannot reliably execute compositional, order-sensitive recipes, highlighting a critical limitation.
The paper introduces CDR-Bench, a benchmark with 3,462 tasks for evaluating LLMs on compositional, order-sensitive data refinement recipes. Experiments on 10+ LLMs show that performance degrades sharply in compositional settings and order-sensitive recipe success collapses, indicating LLMs lack procedural faithfulness.
Data refinement involves executing multi-step recipes over evolving text states, where both composition and execution order of processing operators determine the outcome. While existing benchmarks either isolate text editing or entangle it with code and tool execution, it remains unclear whether LLMs can directly and faithfully execute these compositional, order-sensitive data refinement recipes. To fill this gap, we introduce CDR-Bench, a comprehensive benchmark featuring 3,462 high-quality tasks spanning four real-world data refinement domains and 29 distinct operators. Our benchmark evaluates models across atomic, order-agnostic, and order-sensitive settings, leveraging deterministic reference outputs to enable exact evaluation. Experiments on 10+ state-of-the-art LLMs reveal consistent failure patterns: performance degrades sharply in compositional settings, and order-sensitive recipe success collapses. These findings underline that current LLMs lack the procedural faithfulness required for reliable compositional data refinement.