Identity-Based Patterns in Deep Convolutional Networks: Generative Adversarial Phonology and Reduplication
This work addresses the problem of interpretability and representation learning in neural networks for linguists and AI researchers, though it is incremental as it builds on existing ciwGAN methods.
The paper tackles unsupervised learning of identity-based patterns (reduplication) in speech from raw continuous data using deep convolutional neural networks, specifically the ciwGAN architecture, and demonstrates that the network learns to represent and manipulate these patterns in its latent space, successfully turning unreduplicated forms into reduplicated ones in most cases.
This paper models unsupervised learning of an identity-based pattern (or copying) in speech called reduplication from raw continuous data with deep convolutional neural networks. We use the ciwGAN architecture Beguš (2021a; arXiv:2006.02951) in which learning of meaningful representations in speech emerges from a requirement that the CNNs generate informative data. We propose a technique to wug-test CNNs trained on speech and, based on four generative tests, argue that the network learns to represent an identity-based pattern in its latent space. By manipulating only two categorical variables in the latent space, we can actively turn an unreduplicated form into a reduplicated form with no other substantial changes to the output in the majority of cases. We also argue that the network extends the identity-based pattern to unobserved data. Exploration of how meaningful representations of identity-based patterns emerge in CNNs and how the latent space variables outside of the training range correlate with identity-based patterns in the output has general implications for neural network interpretability.