Spectral Higher-Order Neural Networks
This work addresses a methodological bottleneck for researchers and practitioners in machine learning by enabling higher-order couplings in standard neural networks without requiring hypergraph-structured inputs.
The authors tackled the problem of incorporating higher-order interactions in general-purpose feedforward neural networks, which typically rely on binary interactions, by introducing Spectral Higher-Order Neural Networks (SHONNs) that use spectral attributes to improve stability and parameter scaling.
Neural networks are fundamental tools of modern machine learning. The standard paradigm assumes binary interactions (across feedforward linear passes) between inter-tangled units, organized in sequential layers. Generalized architectures have been also designed that move beyond pairwise interactions, so as to account for higher-order couplings among computing neurons. Higher-order networks are however usually deployed as augmented graph neural networks (GNNs), and, as such, prove solely advantageous in contexts where the input exhibits an explicit hypergraph structure. Here, we present Spectral Higher-Order Neural Networks (SHONNs), a new algorithmic strategy to incorporate higher-order interactions in general-purpose, feedforward, network structures. SHONNs leverages a reformulation of the model in terms of spectral attributes. This allows to mitigate the common stability and parameter scaling problems that come along weighted, higher-order, forward propagations.