Guiding Text-to-Text Privatization by Syntax
This addresses a specific bottleneck in text privatization for applications requiring grammatical fidelity, representing an incremental improvement.
The paper tackles the problem of text-to-text privatization failing to preserve grammatical categories, which leads to surrogate texts dominated by nouns. By guiding substitutions to match grammatical properties, they improved downstream task performance by up to 4.66% while maintaining privacy.
Metric Differential Privacy is a generalization of differential privacy tailored to address the unique challenges of text-to-text privatization. By adding noise to the representation of words in the geometric space of embeddings, words are replaced with words located in the proximity of the noisy representation. Since embeddings are trained based on word co-occurrences, this mechanism ensures that substitutions stem from a common semantic context. Without considering the grammatical category of words, however, this mechanism cannot guarantee that substitutions play similar syntactic roles. We analyze the capability of text-to-text privatization to preserve the grammatical category of words after substitution and find that surrogate texts consist almost exclusively of nouns. Lacking the capability to produce surrogate texts that correlate with the structure of the sensitive texts, we encompass our analysis by transforming the privatization step into a candidate selection problem in which substitutions are directed to words with matching grammatical properties. We demonstrate a substantial improvement in the performance of downstream tasks by up to $4.66\%$ while retaining comparative privacy guarantees.