Not Your Usual Type(s): Data contracts as types across languages and engines
For developers building multi-language data pipelines, this work provides a practical approach to enforce data contracts, reducing runtime errors and improving reliability.
The paper addresses the problem of weak specification at pipeline-node boundaries in composable data systems, proposing a design where data contracts are treated as types. Based on over a year of operating millions of jobs in Bauplan, the authors show that their SDK reduces common production failures by enabling early and consistent schema checking across languages and engines.
Composable data systems promise to let developers combine languages, engines, and catalogs without sacrificing a coherent user experience. In practice, however, pipeline-node boundaries remain weakly specified: transformations exchange tables through schemas that are often checked late, enforced unevenly across languages, and disconnected from the semantics business users care about. Based on over a year of operating millions of jobs in Bauplan, we share the design principles behind our new SDK, which treats data contracts as types for a composable, multi-language lakehouse. Users, whether humans or agents, annotate input and output tables with schema objects that encode column types, constraints, documentation, and lineage; Bauplan then interprets these annotations at different points in the execution lifecycle. We show how this design addresses common production failures, and how an ''everything-as-code'' philosophy enables both deterministic and non-deterministic reasoning over data flows across languages and engines.