Enhanced QKNorm normalization for neural transformers with the Lp norm
This is an incremental improvement for neural network stability in AI models.
The authors tackled the problem of normalizing query and key vectors in Transformers by proposing a generalization of QKNorm using the Lp norm, and experimental results showed it works for a simple problem.
The normalization of query and key vectors is an essential part of the Transformer architecture. It ensures that learning is stable regardless of the scale of these vectors. Some normalization approaches are available. In this preliminary work, a generalization of the QKNorm normalization scheme is proposed. The approach is based on the Lp norm, allowing non-Euclidean norms to be employed. Experimental results demonstrate the suitability of the method for a simple problem.