Interpreting intermediate convolutional layers in unsupervised acoustic word classification
This work addresses the challenge of understanding how unsupervised neural networks classify data, which is incremental as it builds on existing GAN-based architectures for interpretability in speech processing.
The paper tackles the problem of interpreting intermediate layers in unsupervised deep convolutional networks for acoustic word classification by proposing a visualization technique that averages feature maps and uses non-linear regression to infer word distributions, enabling analysis of acoustic properties and phone contrasts across layers.
Understanding how deep convolutional neural networks classify data has been subject to extensive research. This paper proposes a technique to visualize and interpret intermediate layers of unsupervised deep convolutional networks by averaging over individual feature maps in each convolutional layer and inferring underlying distributions of words with non-linear regression techniques. A GAN-based architecture (ciwGAN arXiv:2006.02951) that includes a Generator, a Discriminator, and a classifier was trained on unlabeled sliced lexical items from TIMIT. The training process results in a deep convolutional network that learns to classify words into discrete classes only from the requirement of the Generator to output informative data. This classifier network has no access to the training data -- only to the generated data. We propose a technique to visualize individual convolutional layers in the classifier that yields highly informative time-series data for each convolutional layer and apply it to unobserved test data. Using non-linear regression, we infer underlying distributions for each word which allows us to analyze both absolute values and shapes of individual words at different convolutional layers, as well as perform hypothesis testing on their acoustic properties. The technique also allows us to test individual phone contrasts and how they are represented at each layer.