NL-PAC: Specification Ambiguity and Certified Minimax Risk Floors in LLM-Mediated Supervision
For practitioners using LLMs for supervision, this work provides a method to certify the risk floor due to specification ambiguity, but the guarantee is model-specific and requires external validation for human interpretations.
The paper introduces NL-PAC, a framework that uses a fixed LLM's thresholded decoding to define admissible labels and candidate targets, proving that under target-blind supervision every learner incurs worst-case risk at least half the diameter of the admissible target class. In a Qwen 2.5-3B audit, one prompt yields a positive certificate while a paraphrase and exact-rule controls yield zero.
Large language models increasingly provide labels, evaluations, and feedback for tasks specified in natural language. When a specification admits multiple readings but the supervision channel does not reveal which is operative, additional labels reduce sampling error without resolving the resulting identification problem. We introduce Natural Language PAC (NL-PAC), a framework that uses a fixed model's thresholded decoding law to define admissible labels and candidate targets. The probability that multiple labels are admissible equals the diameter of the pointwise-admissible target class, and under target-blind supervision every learner incurs worst-case risk of at least half this diameter, at every sample size; the exact randomized minimax risk over this class is attained by a data-independent strategy. Finite-sample confidence bounds make these quantities certifiable from held-out unlabeled inputs. In a frozen Qwen~2.5--3B audit, one prespecified prompt yields a positive model-relative certificate, whereas a paraphrase and exact-rule controls yield zero. A held-out bridge audit finds that supplied candidate reading clauses fail the admissibility condition needed to transfer the certificate to coherent readings. The guarantee is specific to the audited model, prompt, threshold, and input distribution; extending it to human interpretations requires external validation.