From Complexity to Clarity: Analytical Expressions of Deep Neural Network Weights via Clifford's Geometric Algebra and Convexity
This provides a novel geometric interpretation of neural network inner workings, potentially aiding researchers in understanding and designing more interpretable models, though it is incremental in applying existing mathematical frameworks to neural networks.
The paper tackles the problem of understanding deep neural network weights by deriving analytical expressions for optimal weights of deep ReLU networks as wedge products of training samples, reducing training to convex optimization over geometric features. The result shows that training involves selecting a small subset of samples via ℓ1 regularization to identify relevant wedge product features, which encode signed volumes of geometric structures in the data.
In this paper, we introduce a novel analysis of neural networks based on geometric (Clifford) algebra and convex optimization. We show that optimal weights of deep ReLU neural networks are given by the wedge product of training samples when trained with standard regularized loss. Furthermore, the training problem reduces to convex optimization over wedge product features, which encode the geometric structure of the training dataset. This structure is given in terms of signed volumes of triangles and parallelotopes generated by data vectors. The convex problem finds a small subset of samples via $\ell_1$ regularization to discover only relevant wedge product features. Our analysis provides a novel perspective on the inner workings of deep neural networks and sheds light on the role of the hidden layers.