Fully tensorial approach to hypercomplex-valued neural networks
This work provides a foundational theoretical framework for hypercomplex-valued neural networks, which could impact researchers in machine learning and AI by offering a unified approach for handling complex data structures, though it appears incremental as it builds on existing constructions for specific algebras.
The paper tackles the problem of enabling neural networks to operate on data defined over arbitrary finite-dimensional algebras by presenting a fully tensorial theoretical framework, which unifies hypercomplex-valued dense and convolutional layers and establishes a tensor-based universal approximation theorem for single-layer perceptrons.
A fully tensorial theoretical framework for hypercomplex-valued neural networks is presented. The proposed approach enables neural network architectures to operate on data defined over arbitrary finite-dimensional algebras. The central observation is that algebra multiplication can be represented by a rank-three tensor, which allows all algebraic operations in neural network layers to be formulated in terms of standard tensor contractions, permutations, and reshaping operations. This tensor-based formulation provides a unified and dimension-independent description of hypercomplex-valued dense and convolutional layers and is directly compatible with modern deep learning libraries supporting optimized tensor operations. The proposed framework recovers existing constructions for four-dimensional algebras as a special case. Within this setting, a tensor-based version of the universal approximation theorem for single-layer hypercomplex-valued perceptrons is established under mild non-degeneracy assumptions on the underlying algebra, thereby providing a rigorous theoretical foundation for the considered class of neural networks.