Curvature Learning for Generalization of Hyperbolic Neural Networks
This work addresses the challenge of optimizing curvature in hyperbolic neural networks for better generalization in hierarchical data tasks, representing an incremental advancement in geometric deep learning.
The paper tackles the problem of suboptimal performance in hyperbolic neural networks (HNNs) due to inappropriate curvatures by deriving a PAC-Bayesian generalization bound that links curvatures to loss landscape smoothness, and proposes a sharpness-aware curvature learning method that improves HNN performance across tasks like classification and few-shot learning, with experimental results showing enhanced generalization.
Hyperbolic neural networks (HNNs) have demonstrated notable efficacy in representing real-world data with hierarchical structures via exploiting the geometric properties of hyperbolic spaces characterized by negative curvatures. Curvature plays a crucial role in optimizing HNNs. Inappropriate curvatures may cause HNNs to converge to suboptimal parameters, degrading overall performance. So far, the theoretical foundation of the effect of curvatures on HNNs has not been developed. In this paper, we derive a PAC-Bayesian generalization bound of HNNs, highlighting the role of curvatures in the generalization of HNNs via their effect on the smoothness of the loss landscape. Driven by the derived bound, we propose a sharpness-aware curvature learning method to smooth the loss landscape, thereby improving the generalization of HNNs. In our method, we design a scope sharpness measure for curvatures, which is minimized through a bi-level optimization process. Then, we introduce an implicit differentiation algorithm that efficiently solves the bi-level optimization by approximating gradients of curvatures. We present the approximation error and convergence analyses of the proposed method, showing that the approximation error is upper-bounded, and the proposed method can converge by bounding gradients of HNNs. Experiments on four settings: classification, learning from long-tailed data, learning from noisy data, and few-shot learning show that our method can improve the performance of HNNs.