A Deep Network with Visual Text Composition Behavior
This addresses the challenge of interpretability in AI for researchers, but it is incremental as it builds on existing neural models.
The paper tackles the problem of understanding compositionality in neural models by proposing a deep network that achieves competitive text classification accuracy and exhibits compositional behavior, where lower layers attend to words and higher layers compose phrases and clauses.
While natural languages are compositional, how state-of-the-art neural models achieve compositionality is still unclear. We propose a deep network, which not only achieves competitive accuracy for text classification, but also exhibits compositional behavior. That is, while creating hierarchical representations of a piece of text, such as a sentence, the lower layers of the network distribute their layer-specific attention weights to individual words. In contrast, the higher layers compose meaningful phrases and clauses, whose lengths increase as the networks get deeper until fully composing the sentence.