Periodic Activation Functions Induce Stationarity
This work addresses reliability and interpretability issues in neural networks for AI practitioners, though it appears incremental by extending known concepts to new activation types.
The paper tackled the problem of neural networks reinforcing hidden data biases by introducing periodic activation functions in Bayesian neural networks to induce stationarity, showing comparable in-domain performance and improved sensitivity to perturbed inputs for out-of-domain detection.
Neural network models are known to reinforce hidden data biases, making them unreliable and difficult to interpret. We seek to build models that `know what they do not know' by introducing inductive biases in the function space. We show that periodic activation functions in Bayesian neural networks establish a connection between the prior on the network weights and translation-invariant, stationary Gaussian process priors. Furthermore, we show that this link goes beyond sinusoidal (Fourier) activations by also covering triangular wave and periodic ReLU activation functions. In a series of experiments, we show that periodic activation functions obtain comparable performance for in-domain data and capture sensitivity to perturbed inputs in deep neural networks for out-of-domain detection.