Circa: Stochastic ReLUs for Private Deep Learning
This addresses the impractical slowdowns in private deep learning for services concerned with user privacy, offering incremental improvements on existing PI methods.
The paper tackles the computational overhead of private inference (PI) by rethinking the ReLU operator, proposing Circa, a set of optimizations that reformulate ReLU as an approximate sign test with novel truncation, resulting in up to 4.7x storage and 3x runtime improvements without impacting accuracy.
The simultaneous rise of machine learning as a service and concerns over user privacy have increasingly motivated the need for private inference (PI). While recent work demonstrates PI is possible using cryptographic primitives, the computational overheads render it impractical. The community is largely unprepared to address these overheads, as the source of slowdown in PI stems from the ReLU operator whereas optimizations for plaintext inference focus on optimizing FLOPs. In this paper we re-think the ReLU computation and propose optimizations for PI tailored to properties of neural networks. Specifically, we reformulate ReLU as an approximate sign test and introduce a novel truncation method for the sign test that significantly reduces the cost per ReLU. These optimizations result in a specific type of stochastic ReLU. The key observation is that the stochastic fault behavior is well suited for the fault-tolerant properties of neural network inference. Thus, we provide significant savings without impacting accuracy. We collectively call the optimizations Circa and demonstrate improvements of up to 4.7x storage and 3x runtime over baseline implementations; we further show that Circa can be used on top of recent PI optimizations to obtain 1.8x additional speedup.