Isotropy, Clusters, and Classifiers
This addresses a theoretical debate in machine learning about embedding space properties, but it is incremental as it clarifies existing results rather than introducing new methods.
The paper shows that enforcing isotropy in embedding spaces is incompatible with the presence of clusters, which harms linear classification performance, as demonstrated mathematically and empirically.
Whether embedding spaces use all their dimensions equally, i.e., whether they are isotropic, has been a recent subject of discussion. Evidence has been accrued both for and against enforcing isotropy in embedding spaces. In the present paper, we stress that isotropy imposes requirements on the embedding space that are not compatible with the presence of clusters -- which also negatively impacts linear classification objectives. We demonstrate this fact both mathematically and empirically and use it to shed light on previous results from the literature.