QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks
This addresses the need for enhanced model performance in various AI tasks, though it appears incremental as it builds on existing architectures.
The paper tackles the problem of limited nonlinearity in deep neural networks by introducing quadratic transformations, achieving clear and substantial performance gains across image classification, text classification, and fine-tuning large-language models.
The combination of linear transformations and non-linear activation functions forms the foundation of most modern deep neural networks, enabling them to approximate highly complex functions. This paper explores the introduction of quadratic transformations to further increase nonlinearity in neural networks, with the aim of enhancing the performance of existing architectures. To reduce parameter complexity and computational complexity, we propose a lightweight quadratic enhancer that uses low-rankness, weight sharing, and sparsification techniques. For a fixed architecture, the proposed approach introduces quadratic interactions between features at every layer, while only adding negligible amounts of additional model parameters and forward computations. We conduct a set of proof-of-concept experiments for the proposed method across three tasks: image classification, text classification, and fine-tuning large-language models. In all tasks, the proposed approach demonstrates clear and substantial performance gains.