Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks
This addresses the challenge of activation function selection for deep learning practitioners, offering a more automated and potentially robust approach, though it is incremental as it builds on existing rational function approximations.
The paper tackles the problem of manually selecting activation functions in deep networks by introducing Padé Activation Units (PAUs), which are flexible parametric rational functions that can approximate common activations and learn new ones, leading to improved predictive performance.
The performance of deep network learning strongly depends on the choice of the non-linear activation function associated with each neuron. However, deciding on the best activation is non-trivial, and the choice depends on the architecture, hyper-parameters, and even on the dataset. Typically these activations are fixed by hand before training. Here, we demonstrate how to eliminate the reliance on first picking fixed activation functions by using flexible parametric rational functions instead. The resulting Padé Activation Units (PAUs) can both approximate common activation functions and also learn new ones while providing compact representations. Our empirical evidence shows that end-to-end learning deep networks with PAUs can increase the predictive performance. Moreover, PAUs pave the way to approximations with provable robustness. https://github.com/ml-research/pau