No Intruder, no Validity: Evaluation Criteria for Privacy-Preserving Text Anonymization
This work tackles the problem of ensuring privacy in shared text data for NLP researchers and practitioners, offering incremental improvements in evaluation standards.
The paper addresses the challenge of evaluating automated text anonymization methods, proposing TILD criteria that assess technical performance, information loss, and human de-anonymization ability to standardize measurement.
For sensitive text data to be shared among NLP researchers and practitioners, shared documents need to comply with data protection and privacy laws. There is hence a growing interest in automated approaches for text anonymization. However, measuring such methods' performance is challenging: missing a single identifying attribute can reveal an individual's identity. In this paper, we draw attention to this problem and argue that researchers and practitioners developing automated text anonymization systems should carefully assess whether their evaluation methods truly reflect the system's ability to protect individuals from being re-identified. We then propose TILD, a set of evaluation criteria that comprises an anonymization method's technical performance, the information loss resulting from its anonymization, and the human ability to de-anonymize redacted documents. These criteria may facilitate progress towards a standardized way for measuring anonymization performance.