Refining Neural Activation Patterns for Layer-Level Concept Discovery in Neural Network-Based Receivers
This work addresses the problem of interpreting neural network behavior for researchers and practitioners, but it is incremental as it builds on existing NAP methodology with specific refinements.
The study tackled the problem of discovering layer-level concepts in neural networks by refining the Neural Activation Pattern (NAP) methodology with improved normalization, distribution estimation, distance metrics, and cluster selection. The result showed that in a radio receiver model, distinct concepts did not emerge, but a continuous activation manifold shaped by Signal-to-Noise Ratio (SNR) was observed, with enhancements improving in-distribution vs. out-of-distribution separation.
Concept discovery in neural networks often targets individual neurons or human-interpretable features, overlooking distributed layer-wide patterns. We study the Neural Activation Pattern (NAP) methodology, which clusters full-layer activation distributions to identify such layer-level concepts. Applied to visual object recognition and radio receiver models, we propose improved normalization, distribution estimation, distance metrics, and varied cluster selection. In the radio receiver model, distinct concepts did not emerge; instead, a continuous activation manifold shaped by Signal-to-Noise Ratio (SNR) was observed -- highlighting SNR as a key learned factor, consistent with classical receiver behavior and supporting physical plausibility. Our enhancements to NAP improved in-distribution vs. out-of-distribution separation, suggesting better generalization and indirectly validating clustering quality. These results underscore the importance of clustering design and activation manifolds in interpreting and troubleshooting neural network behavior.