Beyond The Text: Analysis of Privacy Statements through Syntactic and Semantic Role Labeling
This work addresses the challenge of automating privacy policy analysis for legal and compliance domains, though it is incremental as it adapts existing NLP methods to a new application.
The paper tackled the problem of extracting privacy parameters from legal privacy policies by proposing a new task based on Contextual Integrity, and demonstrated that a combination of syntactic dependency parsing and semantic role labeling achieved the highest accuracy on 36 real-world policies.
This paper formulates a new task of extracting privacy parameters from a privacy policy, through the lens of Contextual Integrity, an established social theory framework for reasoning about privacy norms. Privacy policies, written by lawyers, are lengthy and often comprise incomplete and vague statements. In this paper, we show that traditional NLP tasks, including the recently proposed Question-Answering based solutions, are insufficient to address the privacy parameter extraction problem and provide poor precision and recall. We describe 4 different types of conventional methods that can be partially adapted to address the parameter extraction task with varying degrees of success: Hidden Markov Models, BERT fine-tuned models, Dependency Type Parsing (DP) and Semantic Role Labeling (SRL). Based on a detailed evaluation across 36 real-world privacy policies of major enterprises, we demonstrate that a solution combining syntactic DP coupled with type-specific SRL tasks provides the highest accuracy for retrieving contextual privacy parameters from privacy statements. We also observe that incorporating domain-specific knowledge is critical to achieving high precision and recall, thus inspiring new NLP research to address this important problem in the privacy domain.