Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detection
This work addresses a critical issue for researchers and practitioners in AI security and privacy, as it reveals that synthetic data can fundamentally mislead evaluations of model memorization, potentially affecting broader assessments of data leakage in LLMs.
The paper tackles the problem of misleading membership inference attack (MIA) evaluations when using synthetic data, showing that MIAs act as machine-generated text detectors and incorrectly identify synthetic data as training samples across various models, leading to false conclusions about memorization and data leakage.
Recent work shows membership inference attacks (MIAs) on large language models (LLMs) produce inconclusive results, partly due to difficulties in creating non-member datasets without temporal shifts. While researchers have turned to synthetic data as an alternative, we show this approach can be fundamentally misleading. Our experiments indicate that MIAs function as machine-generated text detectors, incorrectly identifying synthetic data as training samples regardless of the data source. This behavior persists across different model architectures and sizes, from open-source models to commercial ones such as GPT-3.5. Even synthetic text generated by different, potentially larger models is classified as training data by the target model. Our findings highlight a serious concern: using synthetic data in membership evaluations may lead to false conclusions about model memorization and data leakage. We caution that this issue could affect other evaluations using model signals such as loss where synthetic or machine-generated translated data substitutes for real-world samples.