AESPA: Accuracy Preserving Low-degree Polynomial Activation for Fast Private Inference
This work addresses the efficiency problem in private inference for secure machine learning applications, offering a significant speedup with minimal accuracy loss, though it is incremental as it builds on existing polynomial activation methods.
The paper tackles the bottleneck of non-linear activation functions in private inference protocols by proposing AESPA, an accuracy-preserving low-degree polynomial activation, which achieves comparable accuracy to ReLU on models like VGGNet and ResNet while reducing online latency by up to 42.1x and communication cost by up to 28.3x in the Delphi protocol.
Hybrid private inference (PI) protocol, which synergistically utilizes both multi-party computation (MPC) and homomorphic encryption, is one of the most prominent techniques for PI. However, even the state-of-the-art PI protocols are bottlenecked by the non-linear layers, especially the activation functions. Although a standard non-linear activation function can generate higher model accuracy, it must be processed via a costly garbled-circuit MPC primitive. A polynomial activation can be processed via Beaver's multiplication triples MPC primitive but has been incurring severe accuracy drops so far. In this paper, we propose an accuracy preserving low-degree polynomial activation function (AESPA) that exploits the Hermite expansion of the ReLU and basis-wise normalization. We apply AESPA to popular ML models, such as VGGNet, ResNet, and pre-activation ResNet, to show an inference accuracy comparable to those of the standard models with ReLU activation, achieving superior accuracy over prior low-degree polynomial studies. When applied to the all-RELU baseline on the state-of-the-art Delphi PI protocol, AESPA shows up to 42.1x and 28.3x lower online latency and communication cost.