Breaking the Activation Function Bottleneck through Adaptive Parameterization
This work addresses the problem of statistical inefficiency in neural networks for researchers and practitioners in natural language processing, offering a drop-in replacement that improves performance while reducing computational costs.
The paper tackled the inflexibility and inefficiency of standard neural networks by introducing adaptive parameterization methods that condition on input, resulting in a new adaptive LSTM that achieved state-of-the-art results on Penn Treebank and WikiText-2 tasks with fewer parameters and faster convergence.
Standard neural network architectures are non-linear only by virtue of a simple element-wise activation function, making them both brittle and excessively large. In this paper, we consider methods for making the feed-forward layer more flexible while preserving its basic structure. We develop simple drop-in replacements that learn to adapt their parameterization conditional on the input, thereby increasing statistical efficiency significantly. We present an adaptive LSTM that advances the state of the art for the Penn Treebank and WikiText-2 word-modeling tasks while using fewer parameters and converging in less than half as many iterations.