HD-cos Networks: Efficient Neural Architectures for Secure Multi-Party Computation
This addresses the problem of privacy-preserving machine learning for applications with sensitive data, but it is incremental as it builds on existing MPC frameworks with specific optimizations.
The paper tackles the challenge of efficiently training and inferring neural networks under secure multi-party computation (MPC), where operations like ReLU and matrix multiplications are costly due to communication overhead, by proposing HD-cos networks that use cosine activation and Hadamard-Diagonal transformations, demonstrating on public datasets that HD-cos matches the quality of more expensive baselines.
Multi-party computation (MPC) is a branch of cryptography where multiple non-colluding parties execute a well designed protocol to securely compute a function. With the non-colluding party assumption, MPC has a cryptographic guarantee that the parties will not learn sensitive information from the computation process, making it an appealing framework for applications that involve privacy-sensitive user data. In this paper, we study training and inference of neural networks under the MPC setup. This is challenging because the elementary operations of neural networks such as the ReLU activation function and matrix-vector multiplications are very expensive to compute due to the added multi-party communication overhead. To address this, we propose the HD-cos network that uses 1) cosine as activation function, 2) the Hadamard-Diagonal transformation to replace the unstructured linear transformations. We show that both of the approaches enjoy strong theoretical motivations and efficient computation under the MPC setup. We demonstrate on multiple public datasets that HD-cos matches the quality of the more expensive baselines.