The Inhibitor: ReLU and Addition-Based Attention for Efficient Transformers under Fully Homomorphic Encryption on the Torus
This work addresses the problem of enabling efficient, privacy-preserving AI applications on resource-constrained hardware or under homomorphic encryption, though it is incremental as it modifies an existing attention mechanism.
The paper tackles the computational inefficiency of quantized Transformers by replacing dot-product and Softmax-based attention with a ReLU and addition-only mechanism, achieving test set prediction scores comparable to conventional Transformers on four benchmark tasks and suggesting significant computational savings, especially under homomorphic encryption.
To enhance the computational efficiency of quantized Transformers, we replace the dot-product and Softmax-based attention with an alternative mechanism involving addition and ReLU activation only. This side-steps the expansion to double precision often required by matrix multiplication and avoids costly Softmax evaluations but maintains much of the core functionality of conventional dot-product attention. It can enable more efficient execution and support larger quantized Transformer models on resource-constrained hardware or alternative arithmetic systems like homomorphic encryption. Training experiments on four common benchmark tasks show test set prediction scores comparable to those of conventional Transformers with dot-product attention. Our scaling experiments also suggest significant computational savings, both in plaintext and under encryption. In particular, we believe that the ReLU and addition-based attention mechanism examined in this paper may enable privacy-preserving AI applications operating under homomorphic encryption by avoiding the costly multiplication of encrypted variables.