Functional Similarity Metric for Neural Networks: Overcoming Parametric Ambiguity via Activation Region Analysis
For practitioners needing to compare, merge, or interpret neural networks, this work provides a stable similarity metric that overcomes parametric ambiguity, though it is incremental as it combines known techniques (L2 normalization, MinHash, Hungarian algorithm).
The paper addresses representational ambiguity in ReLU networks, where identical functions can arise from different weights due to symmetries. It proposes a metric based on activation region analysis that is robust to weight perturbations, enabling reliable model comparison for merging and interpretability.
As modern deep learning architectures grow in complexity, representational ambiguity emerges as a critical barrier to their interpretability and reliable merging. For ReLU networks, identical functional mappings can be achieved through entirely different weight configurations due to algebraic symmetries: neuron permutation and positive diagonal scaling. Consequently, traditional parameter-based comparison methods exhibit extreme instability to slight weight perturbations during training. This paper proposes a mathematically grounded approach to constructing a stable canonical representation of neural networks and a robust functional similarity metric. We shift focus from comparing raw weights to analyzing the topology of neuron activation regions. The algorithm first eliminates scaling ambiguity via L2-normalization of weight vectors with subsequent layer compensation. Next, discrete approximations of activation regions are generated as binary functional signatures evaluated over a data sample. To overcome the computational bottleneck of comparing large binary vectors, we adapt Locality-Sensitive Hashing, specifically MinHash, providing a fast and statistically precise approximation of the Jaccard index. The final cross-network neuron matching is formulated as a linear sum assignment problem solved via the Hungarian algorithm. We demonstrate theoretically and experimentally that our metric mitigates the neuron "flickering" effect and exhibits exceptional robustness to minor weight perturbations. This framework provides a solid foundation for model merging, transfer learning, objective assessment during pruning, and Explainable AI paradigms.