A visual observation on the geometry of UMAP projections of the difference vectors of antonym and synonym word pair embeddings
This addresses a linguistic problem for NLP researchers, but it is incremental as it focuses on exploratory geometric observations without direct applications.
The paper investigates whether antonym and synonym word pairs can be distinguished geometrically in transformer embeddings by analyzing their difference vectors, finding a curious 'swirl' pattern across models in specific UMAP projections.
Antonyms, or opposites, are sometimes defined as \emph{word pairs that have all of the same contextually relevant properties but one}. Seeing how transformer models seem to encode concepts as directions, this begs the question if one can detect ``antonymity'' in the geometry of the embedding vectors of word pairs, especially based on their difference vectors. Such geometrical studies are then naturally contrasted by comparing antonymic pairs to their opposites; synonyms. This paper started as an exploratory project on the complexity of the systems needed to detect the geometry of the embedding vectors of antonymic word pairs. What we now report is a curious ``swirl'' that appears across embedding models in a somewhat specific projection configuration.