Representing LLMs in Prompt Semantic Task Space
This addresses the problem of efficiently identifying optimal LLMs for users managing expanding model repositories, though it is incremental as it builds on prior representation learning approaches.
The paper tackles the challenge of selecting the best-performing large language model (LLM) for a given task by proposing a training-free method that represents LLMs as linear operators in a semantic task space, achieving competitive or state-of-the-art results in success prediction and model selection tasks.
Large language models (LLMs) achieve impressive results over various tasks, and ever-expanding public repositories contain an abundance of pre-trained models. Therefore, identifying the best-performing LLM for a given task is a significant challenge. Previous works have suggested learning LLM representations to address this. However, these approaches present limited scalability and require costly retraining to encompass additional models and datasets. Moreover, the produced representation utilizes distinct spaces that cannot be easily interpreted. This work presents an efficient, training-free approach to representing LLMs as linear operators within the prompts' semantic task space, thus providing a highly interpretable representation of the models' application. Our method utilizes closed-form computation of geometrical properties and ensures exceptional scalability and real-time adaptability to dynamically expanding repositories. We demonstrate our approach on success prediction and model selection tasks, achieving competitive or state-of-the-art results with notable performance in out-of-sample scenarios.