Map of Encoders -- Mapping Sentence Encoders using Quantum Relative Entropy
This provides a new perspective on the landscape of pre-trained sentence encoders, aiding researchers and practitioners in understanding encoder similarities and performance.
The authors tackled the problem of comparing and visualizing sentence encoders at scale by creating a map of 1101 publicly available encoders using Quantum Relative Entropy, which accurately reflects relationships and predicts downstream task performance.
We propose a method to compare and visualise sentence encoders at scale by creating a map of encoders where each sentence encoder is represented in relation to the other sentence encoders. Specifically, we first represent a sentence encoder using an embedding matrix of a sentence set, where each row corresponds to the embedding of a sentence. Next, we compute the Pairwise Inner Product (PIP) matrix for a sentence encoder using its embedding matrix. Finally, we create a feature vector for each sentence encoder reflecting its Quantum Relative Entropy (QRE) with respect to a unit base encoder. We construct a map of encoders covering 1101 publicly available sentence encoders, providing a new perspective of the landscape of the pre-trained sentence encoders. Our map accurately reflects various relationships between encoders, where encoders with similar attributes are proximally located on the map. Moreover, our encoder feature vectors can be used to accurately infer downstream task performance of the encoders, such as in retrieval and clustering tasks, demonstrating the faithfulness of our map.