Detection of Personal Data in Structured Datasets Using a Large Language Model
This work addresses the need for better personal data detection in structured datasets, which is crucial for privacy and compliance in domains like healthcare, though it is incremental as it builds on existing LLM capabilities.
The paper tackled the problem of detecting personal data in structured datasets by proposing a GPT-4o-based approach that incorporates contextual information, and found that it outperformed existing methods like Microsoft Presidio and CASSED on real-world datasets such as MIMIC-Demo-Ext and Kaggle/OpenML datasets, with strong results attributed to the use of context.
We propose a novel approach for detecting personal data in structured datasets, leveraging GPT-4o, a state-of-the-art Large Language Model. A key innovation of our method is the incorporation of contextual information: in addition to a feature's name and values, we utilize information from other feature names within the dataset as well as the dataset description. We compare our approach to alternative methods, including Microsoft Presidio and CASSED, evaluating them on multiple datasets: DeSSI, a large synthetic dataset, datasets we collected from Kaggle and OpenML as well as MIMIC-Demo-Ext, a real-world dataset containing patient information from critical care units. Our findings reveal that detection performance varies significantly depending on the dataset used for evaluation. CASSED excels on DeSSI, the dataset on which it was trained. Performance on the medical dataset MIMIC-Demo-Ext is comparable across all models, with our GPT-4o-based approach clearly outperforming the others. Notably, personal data detection in the Kaggle and OpenML datasets appears to benefit from contextual information. This is evidenced by the poor performance of CASSED and Presidio (both of which do not utilize the context of the dataset) compared to the strong results of our GPT-4o-based approach. We conclude that further progress in this field would greatly benefit from the availability of more real-world datasets containing personal information.