Clipped Hyperbolic Classifiers Are Super-Hyperbolic Classifiers
This work addresses the limited applicability of HNNs in general classification tasks, making them more competitive across diverse datasets.
The paper tackled the problem of Hyperbolic Neural Networks (HNNs) underperforming on standard benchmarks without clear hierarchies by identifying vanishing gradients as the cause and proposing a simple clipping solution. The result is that clipped HNNs outperform HNNs on hierarchical data and match Euclidean neural networks on benchmarks like MNIST and ImageNet, with improved adversarial robustness and out-of-distribution detection.
Hyperbolic space can naturally embed hierarchies, unlike Euclidean space. Hyperbolic Neural Networks (HNNs) exploit such representational power by lifting Euclidean features into hyperbolic space for classification, outperforming Euclidean neural networks (ENNs) on datasets with known semantic hierarchies. However, HNNs underperform ENNs on standard benchmarks without clear hierarchies, greatly restricting HNNs' applicability in practice. Our key insight is that HNNs' poorer general classification performance results from vanishing gradients during backpropagation, caused by their hybrid architecture connecting Euclidean features to a hyperbolic classifier. We propose an effective solution by simply clipping the Euclidean feature magnitude while training HNNs. Our experiments demonstrate that clipped HNNs become super-hyperbolic classifiers: They are not only consistently better than HNNs which already outperform ENNs on hierarchical data, but also on-par with ENNs on MNIST, CIFAR10, CIFAR100 and ImageNet benchmarks, with better adversarial robustness and out-of-distribution detection.