Specifications: The missing link to making the development of LLM systems an engineering discipline
This work tackles the problem of making LLM system development more reliable and modular for AI engineers and researchers, but it is incremental as it builds on existing advances like structured outputs and process supervision.
The paper addresses the challenge of building modular and robust LLM-based systems by emphasizing the need for precise specifications to describe component behavior, inputs, and outputs, and outlines future research directions to achieve this goal.
Despite the significant strides made by generative AI in just a few short years, its future progress is constrained by the challenge of building modular and robust systems. This capability has been a cornerstone of past technological revolutions, which relied on combining components to create increasingly sophisticated and reliable systems. Cars, airplanes, computers, and software consist of components-such as engines, wheels, CPUs, and libraries-that can be assembled, debugged, and replaced. A key tool for building such reliable and modular systems is specification: the precise description of the expected behavior, inputs, and outputs of each component. However, the generality of LLMs and the inherent ambiguity of natural language make defining specifications for LLM-based components (e.g., agents) both a challenging and urgent problem. In this paper, we discuss the progress the field has made so far-through advances like structured outputs, process supervision, and test-time compute-and outline several future directions for research to enable the development of modular and reliable LLM-based systems through improved specifications.