Reducing ReLU Count for Privacy-Preserving CNN Speedup
This addresses the efficiency bottleneck in privacy-preserving machine learning for applications requiring secure data processing, though it is incremental as it builds on existing cryptographic and CNN methods.
The paper tackles the high communication cost of ReLU operations in privacy-preserving CNNs by proposing shared ReLU decisions across grouped activations, achieving a reduction in ReLU operations by up to three orders of magnitude and cutting communication bandwidth by over 50%.
Privacy-Preserving Machine Learning algorithms must balance classification accuracy with data privacy. This can be done using a combination of cryptographic and machine learning tools such as Convolutional Neural Networks (CNN). CNNs typically consist of two types of operations: a convolutional or linear layer, followed by a non-linear function such as ReLU. Each of these types can be implemented efficiently using a different cryptographic tool. But these tools require different representations and switching between them is time-consuming and expensive. Recent research suggests that ReLU is responsible for most of the communication bandwidth. ReLU is usually applied at each pixel (or activation) location, which is quite expensive. We propose to share ReLU operations. Specifically, the ReLU decision of one activation can be used by others, and we explore different ways to group activations and different ways to determine the ReLU for such a group of activations. Experiments on several datasets reveal that we can cut the number of ReLU operations by up to three orders of magnitude and, as a result, cut the communication bandwidth by more than 50%.