PolyNeuron: Automatic Neuron Discovery via Learned Polyharmonic Spline Activations
This work addresses the under-explored area of automatic neuron design for deep learning practitioners, but it appears incremental as it builds on existing activation function methods with modest gains.
The authors tackled the problem of automated neuron discovery in deep neural networks by proposing PolyNeuron, which learns polyharmonic spline activations, and its relaxed variant PolyNeuron-R to reduce computational complexity. Experiments showed improved or comparable performance on architectures like LeNet-5 and ResNet-20 with datasets such as MNIST and CIFAR10.
Automated deep neural network architecture design has received a significant amount of recent attention. However, this attention has not been equally shared by one of the fundamental building blocks of a deep neural network, the neurons. In this study, we propose PolyNeuron, a novel automatic neuron discovery approach based on learned polyharmonic spline activations. More specifically, PolyNeuron revolves around learning polyharmonic splines, characterized by a set of control points, that represent the activation functions of the neurons in a deep neural network. A relaxed variant of PolyNeuron, which we term PolyNeuron-R, loosens the constraints imposed by PolyNeuron to reduce the computational complexity for discovering the neuron activation functions in an automated manner. Experiments show both PolyNeuron and PolyNeuron-R lead to networks that have improved or comparable performance on multiple network architectures (LeNet-5 and ResNet-20) using different datasets (MNIST and CIFAR10). As such, automatic neuron discovery approaches such as PolyNeuron is a worthy direction to explore.