Position: Ensuring mutual privacy is necessary for effective external evaluation of proprietary AI systems
This addresses the problem of enabling effective external evaluation of proprietary AI systems while protecting privacy for both developers and evaluators, but it is a position paper, so it is incremental in proposing a framework rather than presenting new empirical results.
The paper tackles the challenge of balancing privacy and access in external evaluation of AI systems, arguing that ensuring mutual privacy for both developers and evaluators is essential, and explores potential solutions like cryptographic and hardware-based approaches.
The external evaluation of AI systems is increasingly recognised as a crucial approach for understanding their potential risks. However, facilitating external evaluation in practice faces significant challenges in balancing evaluators' need for system access with AI developers' privacy and security concerns. Additionally, evaluators have reason to protect their own privacy - for example, in order to maintain the integrity of held-out test sets. We refer to the challenge of ensuring both developers' and evaluators' privacy as one of providing mutual privacy. In this position paper, we argue that (i) addressing this mutual privacy challenge is essential for effective external evaluation of AI systems, and (ii) current methods for facilitating external evaluation inadequately address this challenge, particularly when it comes to preserving evaluators' privacy. In making these arguments, we formalise the mutual privacy problem; examine the privacy and access requirements of both model owners and evaluators; and explore potential solutions to this challenge, including through the application of cryptographic and hardware-based approaches.