Embedding principle of homogeneous neural network for classification problem
This work provides insights into network width effects and parameter redundancy in homogeneous neural networks, but it is incremental as it builds on existing theoretical frameworks without broad practical applications.
The paper tackles the relationship between Karush-Kuhn-Tucker (KKT) points in homogeneous neural networks of different widths by introducing the KKT point embedding principle, proving it holds for neuron and channel splitting, and connecting it to gradient flow dynamics to show trajectories remain mapped during training.
In this paper, we study the Karush-Kuhn-Tucker (KKT) points of the associated maximum-margin problem in homogeneous neural networks, including fully-connected and convolutional neural networks. In particular, We investigates the relationship between such KKT points across networks of different widths generated. We introduce and formalize the \textbf{KKT point embedding principle}, establishing that KKT points of a homogeneous network's max-margin problem ($P_Φ$) can be embedded into the KKT points of a larger network's problem ($P_{\tildeΦ}$) via specific linear isometric transformations. We rigorously prove this principle holds for neuron splitting in fully-connected networks and channel splitting in convolutional neural networks. Furthermore, we connect this static embedding to the dynamics of gradient flow training with smooth losses. We demonstrate that trajectories initiated from appropriately mapped points remain mapped throughout training and that the resulting $ω$-limit sets of directions are correspondingly mapped, thereby preserving the alignment with KKT directions dynamically when directional convergence occurs. We conduct several experiments to justify that trajectories are preserved. Our findings offer insights into the effects of network width, parameter redundancy, and the structural connections between solutions found via optimization in homogeneous networks of varying sizes.