Fast and Geometrically Grounded Lorentz Neural Networks
This work addresses a core challenge in hyperbolic representation learning for hierarchical data, offering a more efficient and mathematically grounded approach, though it is incremental in improving existing methods.
The authors tackled the problem of hyperbolic neural networks in the Lorentz model, where existing linear layers cause output norms to scale logarithmically with gradient steps, undermining hyperbolic geometry's advantages. They proposed a new Lorentz linear layer based on distance-to-hyperplane, proving it achieves linear scaling and enabling efficient, geometrically faithful networks.
Hyperbolic space is quickly gaining traction as a promising geometry for hierarchical and robust representation learning. A core open challenge is the development of a mathematical formulation of hyperbolic neural networks that is both efficient and captures the key properties of hyperbolic space. The Lorentz model of hyperbolic space has been shown to enable both fast forward and backward propagation. However, we prove that, with the current formulation of Lorentz linear layers, the hyperbolic norms of the outputs scale logarithmically with the number of gradient descent steps, nullifying the key advantage of hyperbolic geometry. We propose a new Lorentz linear layer grounded in the well-known ``distance-to-hyperplane" formulation. We prove that our formulation results in the usual linear scaling of output hyperbolic norms with respect to the number of gradient descent steps. Our new formulation, together with further algorithmic efficiencies through Lorentzian activation functions and a new caching strategy results in neural networks fully abiding by hyperbolic geometry while simultaneously bridging the computation gap to Euclidean neural networks. Code available at: https://github.com/robertdvdk/hyperbolic-fully-connected.