Parametric Variational Linear Units (PVLUs) in Deep Convolutional Networks
This addresses activation function limitations for deep learning practitioners, offering incremental improvements over ReLU in specific tasks.
The paper tackles the dying neuron problem of ReLU in deep convolutional networks by proposing the Parametric Variational Linear Unit (PVLU), which adds a sinusoidal function with trainable coefficients, resulting in relative error reductions of up to 16.3% on CIFAR-100 and over 10% on CIFAR-10 in various models.
The Rectified Linear Unit is currently a state-of-the-art activation function in deep convolutional neural networks. To combat ReLU's dying neuron problem, we propose the Parametric Variational Linear Unit (PVLU), which adds a sinusoidal function with trainable coefficients to ReLU. Along with introducing nonlinearity and non-zero gradients across the entire real domain, PVLU acts as a mechanism of fine-tuning when implemented in the context of transfer learning. On a simple, non-transfer sequential CNN, PVLU substitution allowed for relative error decreases of 16.3% and 11.3% (without and with data augmentation) on CIFAR-100. PVLU is also tested on transfer learning models. The VGG-16 and VGG-19 models experience relative error reductions of 9.5% and 10.7% on CIFAR-10, respectively, after the substitution of ReLU with PVLU. When training on Gaussian-filtered CIFAR-10 images, similar improvements are noted for the VGG models. Most notably, fine-tuning using PVLU allows for relative error reductions up to and exceeding 10% for near state-of-the-art residual neural network architectures on the CIFAR datasets.