Elementary superexpressive activations
This provides a theoretical foundation for neural network expressivity, potentially enabling more efficient architectures, but it is incremental as it builds on known existence of superexpressive activations.
The paper tackled the problem of finding simple activation functions that allow neural networks to approximate any continuous function with a fixed architecture, proving that the family {sin, arcsin} is superexpressive, while most non-periodic practical activations are not.
We call a finite family of activation functions superexpressive if any multivariate continuous function can be approximated by a neural network that uses these activations and has a fixed architecture only depending on the number of input variables (i.e., to achieve any accuracy we only need to adjust the weights, without increasing the number of neurons). Previously, it was known that superexpressive activations exist, but their form was quite complex. We give examples of very simple superexpressive families: for example, we prove that the family {sin, arcsin} is superexpressive. We also show that most practical activations (not involving periodic functions) are not superexpressive.