Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks
For the database community, this benchmark provides a more comprehensive evaluation of LLMs in database tasks, highlighting limitations of current Text-to-SQL focused benchmarks.
The paper introduces DBLifeBench, the first benchmark evaluating LLMs across five database lifecycle phases (Design, Implementation, Operation, Debugging, Maintenance), and proposes Progressive-Text2SQL, a task using structured reasoning graphs. Evaluation reveals that general-purpose models perform balanced, while specialized Text-to-SQL models suffer from catastrophic forgetting in non-coding phases.
Large Language Models (LLMs) are transforming database interaction paradigms, evolving from simple query translators to autonomous database administrators (DBAs). However, current evaluation benchmarks remain disproportionately fixated on Text-to-SQL tasks, neglecting the holistic Database Lifecycle-from initial schema design to post-deployment maintenance. This narrow focus fails to capture the diverse capabilities required for real-world database management. To bridge this gap, we introduce DBLifeBench, the first benchmark to evaluate LLMs across five critical lifecycle phases: Design, Implementation, Operation, Debugging, and Maintenance. Furthermore, addressing the cognitive mismatch between ambiguous natural language and complex SQL logic, we propose Progressive-Text2SQL, a novel task utilizing structured reasoning graphs to mimic human iterative problem-solving. Our extensive evaluation reveals a critical insight: while general-purpose models demonstrate balanced performance, specialized Text-to-SQL models suffer from ``catastrophic forgetting'' in non-coding phases like design and maintenance. DBLifeBench serves as a foundational step toward evaluating and building true full-stack database intelligence.