Generalization ability and Vulnerabilities to adversarial perturbations: Two sides of the same coin
This addresses the need for more reliable DNNs in high-stakes domains by providing insights into their internal workings, though it is incremental as it applies an existing method (SOM) to analyze DNNs.
The paper tackles the problem of understanding deep neural networks' internal decision-making processes to improve reliability, using self-organizing maps to analyze internal codes and finding that shallow layers have homogeneous codes while deep layers transform them to diverse codes, with evidence linking homogeneous codes to vulnerabilities to adversarial perturbations.
Deep neural networks (DNNs), the agents of deep learning (DL), require a massive number of parallel/sequential operations, which makes it difficult to comprehend them and impedes proper diagnosis. Without better knowledge of DNNs' internal process, deploying DNNs in high-stakes domains may lead to catastrophic failures. Therefore, to build more reliable DNNs/DL, it is imperative that we gain insights into their underlying decision-making process. Here, we use the self-organizing map (SOM) to analyze DL models' internal codes associated with DNNs' decision-making. Our analyses suggest that shallow layers close to the input layer map onto homogeneous codes and that deep layers close to the output layer transform these homogeneous codes in shallow layers to diverse codes. We also found evidence indicating that homogeneous codes may underlie DNNs' vulnerabilities to adversarial perturbations.