Towards Measuring Representational Similarity of Large Language Models
This work addresses the need for reliable similarity metrics in LLMs for practical applications like model selection and security, but it is incremental as it focuses on identifying challenges rather than proposing a new solution.
The paper tackled the problem of measuring representational similarity among large language models (LLMs) with 7B parameters to aid in model selection and detect illegal reuse, finding that some LLMs are substantially different from others and highlighting challenges in using similarity measures to avoid false conclusions.
Understanding the similarity of the numerous released large language models (LLMs) has many uses, e.g., simplifying model selection, detecting illegal model reuse, and advancing our understanding of what makes LLMs perform well. In this work, we measure the similarity of representations of a set of LLMs with 7B parameters. Our results suggest that some LLMs are substantially different from others. We identify challenges of using representational similarity measures that suggest the need of careful study of similarity scores to avoid false conclusions.