PR Product: A Substitute for Inner Product in Neural Networks
This addresses gradient issues in neural networks for researchers and practitioners, offering a drop-in replacement that improves training efficiency and performance across various tasks, though it is incremental as it modifies an existing component.
The authors tackled the problem of diminishing gradient magnitude in neural networks when weight and data vectors align closely, proposing the PR Product as a substitute for the inner product that maintains forward propagation while increasing gradient magnitude. Experiments on CIFAR10, CIFAR100, and MS COCO datasets showed robust enhancement in image classification and competitive performance in image captioning.
In this paper, we analyze the inner product of weight vector w and data vector x in neural networks from the perspective of vector orthogonal decomposition and prove that the direction gradient of w decreases with the angle between them close to 0 or π. We propose the Projection and Rejection Product (PR Product) to make the direction gradient of w independent of the angle and consistently larger than the one in standard inner product while keeping the forward propagation identical. As a reliable substitute for standard inner product, the PR Product can be applied into many existing deep learning modules, so we develop the PR Product version of fully connected layer, convolutional layer and LSTM layer. In static image classification, the experiments on CIFAR10 and CIFAR100 datasets demonstrate that the PR Product can robustly enhance the ability of various state-of-the-art classification networks. On the task of image captioning, even without any bells and whistles, our PR Product version of captioning model can compete or outperform the state-of-the-art models on MS COCO dataset. Code has been made available at:https://github.com/wzn0828/PR_Product.