A Neighbourhood-Aware Differential Privacy Mechanism for Static Word Embeddings
This work addresses privacy concerns in natural language processing for applications using word embeddings, representing an incremental improvement over existing differential privacy methods.
The paper tackled the problem of adding differential privacy to static word embeddings by proposing a Neighbourhood-Aware Differential Privacy (NADP) mechanism that uses word neighbourhoods to minimize noise while ensuring privacy. Experiments showed that NADP consistently outperformed existing DP mechanisms like Laplacian, Gaussian, and Mahalanobis in downstream tasks while providing higher privacy levels.
We propose a Neighbourhood-Aware Differential Privacy (NADP) mechanism considering the neighbourhood of a word in a pretrained static word embedding space to determine the minimal amount of noise required to guarantee a specified privacy level. We first construct a nearest neighbour graph over the words using their embeddings, and factorise it into a set of connected components (i.e. neighbourhoods). We then separately apply different levels of Gaussian noise to the words in each neighbourhood, determined by the set of words in that neighbourhood. Experiments show that our proposed NADP mechanism consistently outperforms multiple previously proposed DP mechanisms such as Laplacian, Gaussian, and Mahalanobis in multiple downstream tasks, while guaranteeing higher levels of privacy.