The 20 questions game to distinguish large language models
This provides a stealthy tool for auditors or copyright owners to detect model leaks, though it is incremental as it builds on existing benchmark datasets and formalizes a known problem.
The authors tackled the problem of distinguishing between large language models (LLMs) in a black-box setting using a small set of binary questions, achieving nearly 100% accuracy within 20 questions with a baseline method and introducing heuristics that cut the required questions in half to discriminate 22 LLMs.
In a parallel with the 20 questions game, we present a method to determine whether two large language models (LLMs), placed in a black-box context, are the same or not. The goal is to use a small set of (benign) binary questions, typically under 20. We formalize the problem and first establish a baseline using a random selection of questions from known benchmark datasets, achieving an accuracy of nearly 100% within 20 questions. After showing optimal bounds for this problem, we introduce two effective questioning heuristics able to discriminate 22 LLMs by using half as many questions for the same task. These methods offer significant advantages in terms of stealth and are thus of interest to auditors or copyright owners facing suspicions of model leaks.