Observations on LLMs for Telecom Domain: Capabilities and Limitations
This work addresses the problem of evaluating LLMs for domain-specific chatbots in telecom, but it is incremental as it applies existing methods to new data without introducing novel techniques.
The paper analyzed the capabilities and limitations of large language models like ChatGPT and LLaMA for building conversational interfaces in the telecom domain, using Cradlepoint's data to compare responses across use-cases such as domain adaptation and robustness, finding insights for data scientists but without reporting specific numerical results.
The landscape for building conversational interfaces (chatbots) has witnessed a paradigm shift with recent developments in generative Artificial Intelligence (AI) based Large Language Models (LLMs), such as ChatGPT by OpenAI (GPT3.5 and GPT4), Google's Bard, Large Language Model Meta AI (LLaMA), among others. In this paper, we analyze capabilities and limitations of incorporating such models in conversational interfaces for the telecommunication domain, specifically for enterprise wireless products and services. Using Cradlepoint's publicly available data for our experiments, we present a comparative analysis of the responses from such models for multiple use-cases including domain adaptation for terminology and product taxonomy, context continuity, robustness to input perturbations and errors. We believe this evaluation would provide useful insights to data scientists engaged in building customized conversational interfaces for domain-specific requirements.