Unified Sparse-Matrix Representations for Diverse Neural Architectures
This work provides a mathematically rigorous substrate for unifying neural architectures, which could benefit researchers and practitioners by simplifying design and leveraging hardware optimization, though it is incremental in its approach.
The authors tackled the problem of diverse neural architectures by introducing a unified matrix-order framework that represents convolutional, recurrent, and self-attention operations as sparse matrix multiplications, achieving performance that matches or exceeds native models across tasks like image classification and language modeling.
Deep neural networks employ specialized architectures for vision, sequential and language tasks, yet this proliferation obscures their underlying commonalities. We introduce a unified matrix-order framework that casts convolutional, recurrent and self-attention operations as sparse matrix multiplications. Convolution is realized via an upper-triangular weight matrix performing first-order transformations; recurrence emerges from a lower-triangular matrix encoding stepwise updates; attention arises naturally as a third-order tensor factorization. We prove algebraic isomorphism with standard CNN, RNN and Transformer layers under mild assumptions. Empirical evaluations on image classification (MNIST, CIFAR-10/100, Tiny ImageNet), time-series forecasting (ETTh1, Electricity Load Diagrams) and language modeling/classification (AG News, WikiText-2, Penn Treebank) confirm that sparse-matrix formulations match or exceed native model performance while converging in comparable or fewer epochs. By reducing architecture design to sparse pattern selection, our matrix perspective aligns with GPU parallelism and leverages mature algebraic optimization tools. This work establishes a mathematically rigorous substrate for diverse neural architectures and opens avenues for principled, hardware-aware network design.