Role Taxonomy of Units in Deep Neural Networks
This work addresses the need for understanding DNN mechanisms and connecting deep learning to neuroscience, but it appears incremental as it builds on existing concepts of unit roles and generalization.
The paper tackled the problem of identifying the roles of units in deep neural networks (DNNs) by introducing a retrieval-of-function test to categorize units into four types based on functional preferences on training and testing sets, and found that the ratios of these categories are highly associated with the generalization ability of DNNs, providing signs for well-generalizing models.
Identifying the role of network units in deep neural networks (DNNs) is critical in many aspects including giving understandings on the mechanisms of DNNs and building basic connections between deep learning and neuroscience. However, there remains unclear on which roles the units in DNNs with different generalization ability could present. To this end, we give role taxonomy of units in DNNs via introducing the retrieval-of-function test, where units are categorized into four types in terms of their functional preference on separately the training set and testing set. We show that ratios of the four categories are highly associated with the generalization ability of DNNs from two distinct perspectives, based on which we give signs of DNNs with well generalization.