Auditing Proprietary Alignment in Large Language Models: A Comparative Framework Without a Ground-Truth Standard
It addresses the challenge of auditing hidden, provider-specific policies in LLMs for external stakeholders like regulators and users.
The paper proposes a statistical framework to detect proprietary alignment in black-box LLMs by comparing their responses to baseline models, enabling auditing without a ground-truth standard. Applied to several cases, it provides a systematic basis for identifying provider-specific alignment.
Large language models (LLMs) are increasingly released and deployed through opaque development and deployment pipelines, enabling model providers to inject intentional, provider-specific policies without officially announcing them. As a result, various models have been reported to generate responses reflecting proprietary rules and organizational interests, leading to censorship or misinformation on controversial topics. However, systematic identification of such alignment remains a fundamental challenge, complicated by the ambiguity of what ``proprietary'' entails in different contexts. In this paper, we propose a statistical framework for detecting proprietary alignment in black-box language models via comparative behavioral analysis. Our approach quantifies systematic deviations between the responses of a target model and those of a reference set of baseline models in a shared semantic space. By evaluating relative behavioral divergence rather than absolute correctness, our framework enables principled auditing under black-box access. Applied to several widely discussed but previously unquantified cases, it provides a systematic and scalable basis for external assessment of provider-specific alignment behavior in large language models.