A Comparative Study of Light-weight Language Models for PII Masking and their Deployment for Real Conversational Texts
This addresses privacy concerns in conversational systems by demonstrating lightweight models can effectively mask PII while mitigating data handling and computational issues of large models, though it is incremental in comparing existing architectures.
The study compared lightweight language models (T5-small and Mistral-Instruct-v0.3) for PII masking in conversational texts, finding they achieve performance comparable to frontier LLMs, with Mistral showing higher F1/recall but higher latency, while T5 offers more controllable outputs and lower cost.
Automated masking of Personally Identifiable Information (PII) is critical for privacy-preserving conversational systems. While current frontier large language models demonstrate strong PII masking capabilities, concerns about data handling and computational costs motivate exploration of whether lightweight models can achieve comparable performance. We compare encoder-decoder and decoder-only architectures by fine-tuning T5-small and Mistral-Instruct-v0.3 on English datasets constructed from the AI4Privacy benchmark. We create different dataset variants to study label standardization and PII representation, covering 24 standardized PII categories and higher-granularity settings. Evaluation using entity-level and character-level metrics, type accuracy, and exact match shows that both lightweight models achieve performance comparable to frontier LLMs for PII masking tasks. Label normalization consistently improves performance across architectures. Mistral achieves higher F1 and recall with greater robustness across PII types but incurs significantly higher generation latency. T5, while less robust in conversational text, offers more controllable structured outputs and lower inference cost, motivating its use in a real-time Discord bot for real-world PII redaction. Evaluation on live messages reveals performance degradation under informal inputs. These results clarify trade-offs between accuracy, robustness, and computational efficiency, demonstrating that lightweight models can provide effective PII masking while addressing data handling concerns associated with frontier LLMs.