The Time is Here for Just-in-Time Systems: Challenges and Opportunities
This work proposes a new paradigm for system building that could drastically reduce development time and improve performance for domain-specific deployments, though it is currently demonstrated only for key-value stores.
The paper introduces Just-in-Time (JIT) Systems, where LLM-based coding agents synthesize entire systems from scratch tailored to specific workloads and constraints. Their Jitskit pipeline synthesizes key-value stores that outperform state-of-the-art systems on all 18 tested specifications, achieving up to 4.6x speedup over the best baseline.
Core systems like key-value stores have historically taken years to build, and are designed to be general so as to amortize cost across deployments, paying a significant performance cost. We argue that LLM-based coding agents now make a different approach tractable: Just-in-Time Systems, in which the entire system is synthesized from scratch, specialized to the environment, workload, and required system properties. We present a JIT system synthesis pipeline, Jitskit, and explore its effectiveness in synthesizing key-value stores from spec cards that span different YCSB workloads, deployment constraints (e.g., compute resources), and system properties (e.g., consistency and durability). Jitskit iteratively refines a system implementation to match the specification against an evolving evaluation test suite. The resulting synthesized systems are performant, beating comparable state-of-the-art systems on 18 of 18 specs tried, by up to 4.6x over the best off-the-shelf baseline on the most favorable spec. Naively running Claude Code either reward-hacks or underperforms Jitskit by up to 5.4x. We discuss the challenges we overcame in building Jitskit and our key takeaways.