Behavioral Fingerprints for LLM Endpoint Stability and Identity
This addresses the need for reliable behavioral monitoring in AI applications, though it is incremental as it builds on existing statistical methods for distribution shift detection.
The paper tackles the problem of monitoring behavioral consistency in LLM endpoints, which traditional reliability metrics fail to capture, by introducing Stability Monitor, a black-box system that detects changes in output distributions over time; in validation, it successfully identified changes due to model updates, quantization, and provider differences.
The consistency of AI-native applications depends on the behavioral consistency of the model endpoints that power them. Traditional reliability metrics such as uptime, latency and throughput do not capture behavioral change, and an endpoint can remain "healthy" while its effective model identity changes due to updates to weights, tokenizers, quantization, inference engines, kernels, caching, routing, or hardware. We introduce Stability Monitor, a black-box stability monitoring system that periodically fingerprints an endpoint by sampling outputs from a fixed prompt set and comparing the resulting output distributions over time. Fingerprints are compared using a summed energy distance statistic across prompts, with permutation-test p-values as evidence of distribution shift aggregated sequentially to detect change events and define stability periods. In controlled validation, Stability Monitor detects changes to model family, version, inference stack, quantization, and behavioral parameters. In real-world monitoring of the same model hosted by multiple providers, we observe substantial provider-to-provider and within-provider stability differences.