All Roads Lead to Rome? Exploring the Invariance of Transformers' Representations
This work addresses a fundamental reliability issue in interpretability research for NLP, though it is incremental as it builds on existing bijective alignment methods.
The paper tackles the problem of whether transformer models learn invariant representation spaces across different random seeds, proposing the Bijection Hypothesis and BERT-INN, an invertible neural network method, which outperforms existing bijective methods like CCA in aligning BERT embeddings.
Transformer models bring propelling advances in various NLP tasks, thus inducing lots of interpretability research on the learned representations of the models. However, we raise a fundamental question regarding the reliability of the representations. Specifically, we investigate whether transformers learn essentially isomorphic representation spaces, or those that are sensitive to the random seeds in their pretraining process. In this work, we formulate the Bijection Hypothesis, which suggests the use of bijective methods to align different models' representation spaces. We propose a model based on invertible neural networks, BERT-INN, to learn the bijection more effectively than other existing bijective methods such as the canonical correlation analysis (CCA). We show the advantage of BERT-INN both theoretically and through extensive experiments, and apply it to align the reproduced BERT embeddings to draw insights that are meaningful to the interpretability research. Our code is at https://github.com/twinkle0331/BERT-similarity.