What Do Deep Nets Learn? Class-wise Patterns Revealed in the Input Space
This provides a tool for better understanding DNN knowledge, which is incremental as it builds on existing visualization methods.
The paper tackles the problem of understanding what deep neural networks learn by proposing a method to visualize class-wise patterns in the input space for image classification, showing that models learn abstract shapes and textures in natural settings, suspicious patterns in backdoored settings, and simplified shapes in adversarial settings.
Deep neural networks (DNNs) are increasingly deployed in different applications to achieve state-of-the-art performance. However, they are often applied as a black box with limited understanding of what knowledge the model has learned from the data. In this paper, we focus on image classification and propose a method to visualize and understand the class-wise knowledge (patterns) learned by DNNs under three different settings including natural, backdoor and adversarial. Different to existing visualization methods, our method searches for a single predictive pattern in the pixel space to represent the knowledge learned by the model for each class. Based on the proposed method, we show that DNNs trained on natural (clean) data learn abstract shapes along with some texture, and backdoored models learn a suspicious pattern for the backdoored class. Interestingly, the phenomenon that DNNs can learn a single predictive pattern for each class indicates that DNNs can learn a backdoor even from clean data, and the pattern itself is a backdoor trigger. In the adversarial setting, we show that adversarially trained models tend to learn more simplified shape patterns. Our method can serve as a useful tool to better understand the knowledge learned by DNNs on different datasets under different settings.