Log Probability Tracking of LLM APIs
This addresses the issue of unmonitored model updates in LLM APIs, crucial for downstream applications and research, though it is incremental as it builds on existing audit methods.
The paper tackled the problem of monitoring LLM API consistency for reliability and reproducibility by proposing a cost-effective method using log probabilities, which can detect changes as small as one step of fine-tuning with 1,000x lower cost than existing methods.
When using an LLM through an API provider, users expect the served model to remain consistent over time, a property crucial for the reliability of downstream applications and the reproducibility of research. Existing audit methods are too costly to apply at regular time intervals to the wide range of available LLM APIs. This means that model updates are left largely unmonitored in practice. In this work, we show that while LLM log probabilities (logprobs) are usually non-deterministic, they can still be used as the basis for cost-effective continuous monitoring of LLM APIs. We apply a simple statistical test based on the average value of each token logprob, requesting only a single token of output. This is enough to detect changes as small as one step of fine-tuning, making this approach more sensitive than existing methods while being 1,000x cheaper. We introduce the TinyChange benchmark as a way to measure the sensitivity of audit methods in the context of small, realistic model changes.