Where We Have Arrived in Proving the Emergence of Sparse Symbolic Concepts in AI Models
It addresses the theoretical understanding of symbolic reasoning in AI models, which is incremental as it builds on existing conditions to formalize emergence.
This study tackled the problem of proving the emergence of sparse symbolic concepts in deep neural networks by establishing three common conditions under which DNNs encode only a small number of sparse interactions, showing that inference scores on many masked samples can be mimicked by just a few interactions.
This study aims to prove the emergence of symbolic concepts (or more precisely, sparse primitive inference patterns) in well-trained deep neural networks (DNNs). Specifically, we prove the following three conditions for the emergence. (i) The high-order derivatives of the network output with respect to the input variables are all zero. (ii) The DNN can be used on occluded samples and when the input sample is less occluded, the DNN will yield higher confidence. (iii) The confidence of the DNN does not significantly degrade on occluded samples. These conditions are quite common, and we prove that under these conditions, the DNN will only encode a relatively small number of sparse interactions between input variables. Moreover, we can consider such interactions as symbolic primitive inference patterns encoded by a DNN, because we show that inference scores of the DNN on an exponentially large number of randomly masked samples can always be well mimicked by numerical effects of just a few interactions.