Compliance Checking with NLI: Privacy Policies vs. Regulations
This addresses the challenge of efficiently verifying compliance for companies and regulators, but it is incremental as it applies existing NLI methods to a new domain.
The paper tackled the problem of automatically checking privacy policies for legal inconsistencies by using Natural Language Inference (NLI) techniques to compare regulations against policy sections, finding that a model trained on the MNLI dataset performed better in real-world tasks despite lower test accuracy.
A privacy policy is a document that states how a company intends to handle and manage their customers' personal data. One of the problems that arises with these privacy policies is that their content might violate data privacy regulations. Because of the enormous number of privacy policies that exist, the only realistic way to check for legal inconsistencies in all of them is through an automated method. In this work, we use Natural Language Inference (NLI) techniques to compare privacy regulations against sections of privacy policies from a selection of large companies. Our NLI model uses pre-trained embeddings, along with BiLSTM in its attention mechanism. We tried two versions of our model: one that was trained on the Stanford Natural Language Inference (SNLI) and the second on the Multi-Genre Natural Language Inference (MNLI) dataset. We found that our test accuracy was higher on our model trained on the SNLI, but when actually doing NLI tasks on real world privacy policies, the model trained on MNLI generalized and performed much better.