Multilingual, Multi-scale and Multi-layer Visualization of Intermediate Representations
This provides a tool for researchers and practitioners to better understand and debug sequence models, though it is incremental as it builds on existing visualization approaches.
The authors tackled the problem of interpreting intermediate layer representations in sequence-based architectures by introducing a web-based visualization tool that shows representations at sentence and token levels, demonstrating its utility through three use cases including gender bias analysis and multilingual machine translation.
The main alternatives nowadays to deal with sequences are Recurrent Neural Networks (RNN), Convolutional Neural Networks (CNN) architectures and the Transformer. In this context, RNN's, CNN's and Transformer have most commonly been used as an encoder-decoder architecture with multiple layers in each module. Far beyond this, these architectures are the basis for the contextual word embeddings which are revolutionizing most natural language downstream applications. However, intermediate layer representations in sequence-based architectures can be difficult to interpret. To make each layer representation within these architectures more accessible and meaningful, we introduce a web-based tool that visualizes them both at the sentence and token level. We present three use cases. The first analyses gender issues in contextual word embeddings. The second and third are showing multilingual intermediate representations for sentences and tokens and the evolution of these intermediate representations along the multiple layers of the decoder and in the context of multilingual machine translation.