LLM-42: Enabling Determinism in LLM Inference with Verified Speculation
This addresses a critical reliability issue for users who need reproducible LLM outputs in applications like scientific computing or debugging, though it is an incremental improvement on existing speculation techniques.
The paper tackles the problem of non-deterministic outputs in LLM inference caused by floating-point non-associativity and dynamic batching, and presents LLM-42, a scheduling-based approach that enables determinism with minimal overhead, achieving up to 95% of the throughput of non-deterministic baselines while ensuring consistent outputs.
In LLM inference, the same prompt may yield different outputs across different runs. At the system level, this non-determinism arises from floating-point non-associativity combined with dynamic batching and GPU kernels whose reduction orders vary with batch size. A straightforward way to eliminate non-determinism is to disable dynamic batching during inference, but doing so severely degrades throughput. Another approach is to make kernels batch-invariant; however, this tightly couples determinism to kernel design, requiring new implementations. This coupling also imposes fixed runtime overheads, regardless of how much of the workload actually requires determinism. Inspired by ideas from speculative decoding, we present LLM-42, a scheduling-based approach to enable determinism in LLM inference. Our key observation is that if a sequence is in a consistent state, the next emitted token is likely to be consistent even with dynamic batching. Moreover, most GPU kernels use shape-consistent reductions. Leveraging these insights, LLM-42 decodes tokens using a non-deterministic fast path and enforces determinism via a lightweight verify-rollback loop. The verifier replays candidate tokens under a fixed-shape reduction schedule, commits those that are guaranteed to be consistent across runs, and rolls back those violating determinism. LLM-42 mostly re-uses existing kernels unchanged and incurs overhead only in proportion to the traffic that requires determinism.