Membership Inference on LLMs in the Wild
This addresses privacy auditing for LLMs, providing a tool for researchers and practitioners concerned with data leakage, though it is incremental as it builds on existing MIA techniques.
The paper tackles the problem of membership inference attacks on large language models in black-box settings where only generated text is available, proposing SimMIA which achieves state-of-the-art results and rivals baselines using internal model information.
Membership Inference Attacks (MIAs) act as a crucial auditing tool for the opaque training data of Large Language Models (LLMs). However, existing techniques predominantly rely on inaccessible model internals (e.g., logits) or suffer from poor generalization across domains in strict black-box settings where only generated text is available. In this work, we propose SimMIA, a robust MIA framework tailored for this text-only regime by leveraging an advanced sampling strategy and scoring mechanism. Furthermore, we present WikiMIA-25, a new benchmark curated to evaluate MIA performance on modern proprietary LLMs. Experiments demonstrate that SimMIA achieves state-of-the-art results in the black-box setting, rivaling baselines that exploit internal model information.