Language Models as a Service: Overview of a New Paradigm and its Challenges
It addresses evaluation problems for researchers and practitioners using closed-off language models, but is incremental as it synthesizes existing knowledge rather than proposing new methods.
The paper identifies challenges in evaluating, benchmarking, and testing proprietary language models accessed via APIs, analyzing how lack of information impedes accessibility, replicability, reliability, and trustworthiness, and provides recommendations and an overview of current LMaaS offerings.
Some of the most powerful language models currently are proprietary systems, accessible only via (typically restrictive) web or software programming interfaces. This is the Language-Models-as-a-Service (LMaaS) paradigm. In contrast with scenarios where full model access is available, as in the case of open-source models, such closed-off language models present specific challenges for evaluating, benchmarking, and testing them. This paper has two goals: on the one hand, we delineate how the aforementioned challenges act as impediments to the accessibility, replicability, reliability, and trustworthiness of LMaaS. We systematically examine the issues that arise from a lack of information about language models for each of these four aspects. We conduct a detailed analysis of existing solutions and put forth a number of considered recommendations, and highlight the directions for future advancements. On the other hand, it serves as a comprehensive resource for existing knowledge on current, major LMaaS, offering a synthesized overview of the licences and capabilities their interfaces offer.