KITINet: Kinetics Theory Inspired Network Architectures with PDE Simulation Approaches
This work addresses the problem of improving neural network design for researchers and practitioners by offering a physics-inspired approach, though it appears incremental as it builds on existing residual connection frameworks.
The paper tackled the heuristic design of residual connections in neural networks by introducing KITINet, a novel architecture inspired by non-equilibrium particle dynamics and PDE simulation, which achieved consistent improvements over classic baselines on tasks like image and text classification with negligible FLOPs increase.
Despite the widely recognized success of residual connections in modern neural networks, their design principles remain largely heuristic. This paper introduces KITINet (Kinetics Theory Inspired Network), a novel architecture that reinterprets feature propagation through the lens of non-equilibrium particle dynamics and partial differential equation (PDE) simulation. At its core, we propose a residual module that models feature updates as the stochastic evolution of a particle system, numerically simulated via a discretized solver for the Boltzmann transport equation (BTE). This formulation mimics particle collisions and energy exchange, enabling adaptive feature refinement via physics-informed interactions. Additionally, we reveal that this mechanism induces network parameter condensation during training, where parameters progressively concentrate into a sparse subset of dominant channels. Experiments on scientific computation (PDE operator), image classification (CIFAR-10/100), and text classification (IMDb/SNLI) show consistent improvements over classic network baselines, with negligible increase of FLOPs.