Revisiting Generalization Power of a DNN in Terms of Symbolic Interactions
This provides a new theoretical perspective on generalization in deep learning, which is a foundational problem for machine learning researchers.
The paper tackles the problem of understanding the generalization power of deep neural networks by analyzing interactions, finding that generalizable interactions follow a decay-shaped distribution and non-generalizable ones follow a spindle-shaped distribution, and verifying this theory matches real interactions in experiments.
This paper aims to analyze the generalization power of deep neural networks (DNNs) from the perspective of interactions. Unlike previous analysis of a DNN's generalization power in a highdimensional feature space, we find that the generalization power of a DNN can be explained as the generalization power of the interactions. We found that the generalizable interactions follow a decay-shaped distribution, while non-generalizable interactions follow a spindle-shaped distribution. Furthermore, our theory can effectively disentangle these two types of interactions from a DNN. We have verified that our theory can well match real interactions in a DNN in experiments.