Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decoding
This addresses privacy and security concerns for users and developers of large language models by improving membership inference, though it is incremental as it builds on existing methods by incorporating both member and non-member contexts.
The paper tackles the problem of detecting pre-training data in large language models to mitigate privacy and security risks, achieving state-of-the-art performance on the WikiMIA benchmark through a contrastive decoding approach that leverages subtle distributional shifts.
The training data in large language models is key to their success, but it also presents privacy and security risks, as it may contain sensitive information. Detecting pre-training data is crucial for mitigating these concerns. Existing methods typically analyze target text in isolation or solely with non-member contexts, overlooking potential insights from simultaneously considering both member and non-member contexts. While previous work suggested that member contexts provide little information due to the minor distributional shift they induce, our analysis reveals that these subtle shifts can be effectively leveraged when contrasted with non-member contexts. In this paper, we propose Con-ReCall, a novel approach that leverages the asymmetric distributional shifts induced by member and non-member contexts through contrastive decoding, amplifying subtle differences to enhance membership inference. Extensive empirical evaluations demonstrate that Con-ReCall achieves state-of-the-art performance on the WikiMIA benchmark and is robust against various text manipulation techniques.