Discovery of Natural Language Concepts in Individual Units of CNNs
This work addresses the interpretability challenge for researchers and practitioners using CNNs in NLP, providing insights into how these models represent language, though it is incremental in advancing understanding rather than introducing a new paradigm.
The study tackled the problem of interpreting deep convolutional networks in natural language tasks by showing that individual units respond selectively to specific linguistic concepts like morphemes and words, rather than uninterpretable patterns, and proposed a concept alignment method for quantitative analysis across multiple datasets and architectures.
Although deep convolutional networks have achieved improved performance in many natural language tasks, they have been treated as black boxes because they are difficult to interpret. Especially, little is known about how they represent language in their intermediate layers. In an attempt to understand the representations of deep convolutional networks trained on language tasks, we show that individual units are selectively responsive to specific morphemes, words, and phrases, rather than responding to arbitrary and uninterpretable patterns. In order to quantitatively analyze such an intriguing phenomenon, we propose a concept alignment method based on how units respond to the replicated text. We conduct analyses with different architectures on multiple datasets for classification and translation tasks and provide new insights into how deep models understand natural language.