Element-wise Modulation of Random Matrices for Efficient Neural Layers
This provides a stable and computationally efficient solution for scaling and deploying neural networks in resource-limited settings, addressing a common bottleneck in deep learning.
The paper tackles the memory and computational overhead of fully connected layers in deep neural networks by proposing the Parametrized Random Projection (PRP) layer, which uses a fixed random matrix with learnable element-wise parameters to reduce trainable parameters to a linear scale while maintaining reliable accuracy across benchmarks.
Fully connected layers are a primary source of memory and computational overhead in deep neural networks due to their dense, often redundant parameterization. While various compression techniques exist, they frequently introduce complex engineering trade-offs or degrade model performance. We propose the Parametrized Random Projection (PRP) layer, a novel approach that decouples feature mixing from adaptation by utilizing a fixed random matrix modulated by lightweight, learnable element-wise parameters. This architecture drastically reduces the trainable parameter count to a linear scale while retaining reliable accuracy across various benchmarks. The design serves as a stable, computationally efficient solution for architectural scaling and deployment in resource-limited settings.