Why are hyperbolic neural networks effective? A study on hierarchical representation capability
This work addresses a foundational flaw in the motivation for hyperbolic neural networks, providing a benchmark and analysis that could impact research in hierarchical data representation.
The study found that hyperbolic neural networks (HNNs) cannot achieve the theoretical optimal embedding for hierarchical representation, and that enhancing this capability through pre-training strategies significantly improves their performance on downstream tasks.
Hyperbolic Neural Networks (HNNs), operating in hyperbolic space, have been widely applied in recent years, motivated by the existence of an optimal embedding in hyperbolic space that can preserve data hierarchical relationships (termed Hierarchical Representation Capability, HRC) more accurately than Euclidean space. However, there is no evidence to suggest that HNNs can achieve this theoretical optimal embedding, leading to much research being built on flawed motivations. In this paper, we propose a benchmark for evaluating HRC and conduct a comprehensive analysis of why HNNs are effective through large-scale experiments. Inspired by the analysis results, we propose several pre-training strategies to enhance HRC and improve the performance of downstream tasks, further validating the reliability of the analysis. Experiments show that HNNs cannot achieve the theoretical optimal embedding. The HRC is significantly affected by the optimization objectives and hierarchical structures, and enhancing HRC through pre-training strategies can significantly improve the performance of HNNs.