Topology-Aware Activation Functions in Neural Networks
This addresses the problem of enhancing neural network performance in scenarios with low-dimensional layers, offering a domain-specific improvement that is incremental in nature.
This study tackled the problem of neural networks' limited ability to manipulate data topology by proposing novel activation functions like SmoothSplit and ParametricSplit, which introduce topology 'cutting' capabilities. The result showed that ParametricSplit outperforms traditional activations in low-dimensional settings while maintaining competitive performance in higher-dimensional ones.
This study explores novel activation functions that enhance the ability of neural networks to manipulate data topology during training. Building on the limitations of traditional activation functions like $\mathrm{ReLU}$, we propose $\mathrm{SmoothSplit}$ and $\mathrm{ParametricSplit}$, which introduce topology "cutting" capabilities. These functions enable networks to transform complex data manifolds effectively, improving performance in scenarios with low-dimensional layers. Through experiments on synthetic and real-world datasets, we demonstrate that $\mathrm{ParametricSplit}$ outperforms traditional activations in low-dimensional settings while maintaining competitive performance in higher-dimensional ones. Our findings highlight the potential of topology-aware activation functions in advancing neural network architectures. The code is available via https://github.com/Snopoff/Topology-Aware-Activations.