Enhancing Clinical Models with Pseudo Data for De-identification
This addresses privacy concerns in clinical AI by enhancing de-identification models, though it appears incremental as it builds on existing pretraining approaches.
The paper tackles the problem of training clinical foundation models on redacted text for privacy, finding that using realistic pseudo text replacements significantly improves performance on de-identification tasks, with methods outperforming previous baselines.
Many models are pretrained on redacted text for privacy reasons. Clinical foundation models are often trained on de-identified text, which uses special syntax (masked) text in place of protected health information. Even though these models have increased in popularity, there has been little effort in understanding the effects of training them on redacted text. In this work, we pretrain several encoder-only models on a dataset that contains redacted text and a version with replaced realistic pseudo text. We then fine-tuned models for the protected health information de-identification task and show how our methods significantly outperform previous baselines. The contributions of this work include: a) our novel, and yet surprising findings with training recommendations, b) redacted text replacements used to produce the pseudo dataset, c) pretrained embeddings and fine-tuned task specific models, and d) freely available pseudo training dataset generation and model source code used in our experiments.