Similarity Analysis of Contextual Word Representation Models
This work provides insights into model design factors for researchers in natural language processing, but it is incremental as it builds on existing similarity analysis methods.
The paper investigates the similarity of internal representations and attention across different contextual word representation models, finding that while different architectures have similar representations, individual neurons differ, and higher layers are more affected by fine-tuning on downstream tasks.
This paper investigates contextual word representation models from the lens of similarity analysis. Given a collection of trained models, we measure the similarity of their internal representations and attention. Critically, these models come from vastly different architectures. We use existing and novel similarity measures that aim to gauge the level of localization of information in the deep models, and facilitate the investigation of which design factors affect model similarity, without requiring any external linguistic annotation. The analysis reveals that models within the same family are more similar to one another, as may be expected. Surprisingly, different architectures have rather similar representations, but different individual neurons. We also observed differences in information localization in lower and higher layers and found that higher layers are more affected by fine-tuning on downstream tasks.