Disparate Impact on Group Accuracy of Linearization for Private Inference
This work addresses fairness issues in privacy-preserving machine learning, highlighting a trade-off between computational efficiency and group accuracy, which is incremental as it builds on existing linearization methods.
The paper tackles the problem of privacy-preserving inference by linearizing ReLU activations to reduce computational costs, but finds that this disproportionately decreases accuracy for minority groups compared to majority groups, with mitigation shown through a fine-tuning procedure.
Ensuring privacy-preserving inference on cryptographically secure data is a well-known computational challenge. To alleviate the bottleneck of costly cryptographic computations in non-linear activations, recent methods have suggested linearizing a targeted portion of these activations in neural networks. This technique results in significantly reduced runtimes with often negligible impacts on accuracy. In this paper, we demonstrate that such computational benefits may lead to increased fairness costs. Specifically, we find that reducing the number of ReLU activations disproportionately decreases the accuracy for minority groups compared to majority groups. To explain these observations, we provide a mathematical interpretation under restricted assumptions about the nature of the decision boundary, while also showing the prevalence of this problem across widely used datasets and architectures. Finally, we show how a simple procedure altering the fine-tuning step for linearized models can serve as an effective mitigation strategy.