On Isotropy Calibration of Transformers
This work addresses the relevance of isotropy calibration for transformer models, suggesting it may be unnecessary, which is incremental as it builds on prior studies of embedding space properties.
The paper tackles the problem of isotropy calibration in transformer models, finding through empirical evaluation that state-of-the-art calibration methods do not consistently improve performance across models and tasks, supporting the idea that transformers do not benefit from such calibration due to local isotropy.
Different studies of the embedding space of transformer models suggest that the distribution of contextual representations is highly anisotropic - the embeddings are distributed in a narrow cone. Meanwhile, static word representations (e.g., Word2Vec or GloVe) have been shown to benefit from isotropic spaces. Therefore, previous work has developed methods to calibrate the embedding space of transformers in order to ensure isotropy. However, a recent study (Cai et al. 2021) shows that the embedding space of transformers is locally isotropic, which suggests that these models are already capable of exploiting the expressive capacity of their embedding space. In this work, we conduct an empirical evaluation of state-of-the-art methods for isotropy calibration on transformers and find that they do not provide consistent improvements across models and tasks. These results support the thesis that, given the local isotropy, transformers do not benefit from additional isotropy calibration.