Understanding Activation Patterns in Artificial Neural Networks by Exploring Stochastic Processes
This work provides a novel mathematical framework for analyzing neural network behavior, with potential applications in theoretical simulations, pruning, and transfer learning, though it is incremental in applying neuroscience techniques to artificial networks.
The paper tackled the problem of understanding activation patterns in artificial neural networks by modeling them as stochastic processes, specifically using Poisson distributions to analyze activation frequency, and found stable indicators of memorization during learning, such as differences in Mean Firing Rate and Fano Factor across network types.
To gain a deeper understanding of the behavior and learning dynamics of (deep) artificial neural networks, it is valuable to employ mathematical abstractions and models. These tools provide a simplified perspective on network performance and facilitate systematic investigations through simulations. In this paper, we propose utilizing the framework of stochastic processes, which has been underutilized thus far. Our approach models activation patterns of thresholded nodes in (deep) artificial neural networks as stochastic processes. We focus solely on activation frequency, leveraging neuroscience techniques used for real neuron spike trains. During a classification task, we extract spiking activity and use an arrival process following the Poisson distribution. We examine observed data from various artificial neural networks in image recognition tasks, fitting the proposed model's assumptions. Through this, we derive parameters describing activation patterns in each network. Our analysis covers randomly initialized, generalizing, and memorizing networks, revealing consistent differences across architectures and training sets. Calculating Mean Firing Rate, Mean Fano Factor, and Variances, we find stable indicators of memorization during learning, providing valuable insights into network behavior. The proposed model shows promise in describing activation patterns and could serve as a general framework for future investigations. It has potential applications in theoretical simulations, pruning, and transfer learning.