Knowledge Consistency between Neural Networks and Beyond
This work addresses the need for better interpretability and refinement of neural networks, though it appears incremental as it builds on existing concepts like knowledge distillation without introducing a new paradigm.
The paper tackles the problem of analyzing knowledge consistency between pre-trained deep neural networks by proposing a generic definition and a task-agnostic method to disentangle consistent feature components, with preliminary experiments showing it can diagnose representations, explain techniques like knowledge distillation, and refine networks to boost performance.
This paper aims to analyze knowledge consistency between pre-trained deep neural networks. We propose a generic definition for knowledge consistency between neural networks at different fuzziness levels. A task-agnostic method is designed to disentangle feature components, which represent the consistent knowledge, from raw intermediate-layer features of each neural network. As a generic tool, our method can be broadly used for different applications. In preliminary experiments, we have used knowledge consistency as a tool to diagnose representations of neural networks. Knowledge consistency provides new insights to explain the success of existing deep-learning techniques, such as knowledge distillation and network compression. More crucially, knowledge consistency can also be used to refine pre-trained networks and boost performance.