Developing Training Procedures for Piecewise-linear Spline Activation Functions in Neural Networks
This work addresses the need for more efficient and accurate neural networks, though it involves increased complexity and latency, making it incremental in nature.
The paper tackled the problem of optimizing activation function shapes in neural networks to improve parameter efficiency and accuracy, achieving up to 94% lower error rates in FNNs and 51% lower rates in CNNs compared to ReLU-based models.
Activation functions in neural networks are typically selected from a set of empirically validated, commonly used static functions such as ReLU, tanh, or sigmoid. However, by optimizing the shapes of a network's activation functions, we can train models that are more parameter-efficient and accurate by assigning more optimal activations to the neurons. In this paper, I present and compare 9 training methodologies to explore dual-optimization dynamics in neural networks with parameterized linear B-spline activation functions. The experiments realize up to 94% lower end model error rates in FNNs and 51% lower rates in CNNs compared to traditional ReLU-based models. These gains come at the cost of additional development and training complexity as well as end model latency.