On the Mathematical Relationship Between Layer Normalization and Dynamic Activation Functions
This work provides a theoretical link between layer normalization and dynamic activation functions, offering a more precise alternative for neural network design, though it is incremental in nature.
The paper tackles the lack of theoretical foundation for dynamic activation functions like Dynamic Tanh by deriving it from RMSNorm and proposing an exact counterpart, DyISRU, which more accurately reproduces normalization effects on outliers.
Layer normalization (LN) is an essential component of modern neural networks. While many alternative techniques have been proposed, none of them have succeeded in replacing LN so far. The latest suggestion in this line of research is a dynamic activation function called Dynamic Tanh (DyT). Although it is empirically well-motivated and appealing from a practical point of view, it lacks a theoretical foundation. In this work, we shed light on the mathematical relationship between LN and dynamic activation functions. In particular, we derive DyT from the LN variant RMSNorm, and show that a well-defined decoupling in derivative space as well as an approximation are needed to do so. By applying the same decoupling procedure directly in function space, we are able to omit the approximation and obtain the exact element-wise counterpart of RMSNorm, which we call Dynamic Inverse Square Root Unit (DyISRU). We demonstrate numerically that DyISRU reproduces the normalization effect on outliers more accurately than DyT does.