A State-of-the-practice Release-readiness Checklist for Generative AI-based Software Products
This addresses the challenge for software practitioners in deploying LLM-based applications, though it is incremental as it builds on existing grey literature.
The paper tackles the problem of determining release readiness for generative AI-based software products by developing a checklist to guide practitioners in evaluating performance, monitoring, and deployment strategies, aiming to enhance reliability and effectiveness.
This paper investigates the complexities of integrating Large Language Models (LLMs) into software products, with a focus on the challenges encountered for determining their readiness for release. Our systematic review of grey literature identifies common challenges in deploying LLMs, ranging from pre-training and fine-tuning to user experience considerations. The study introduces a comprehensive checklist designed to guide practitioners in evaluating key release readiness aspects such as performance, monitoring, and deployment strategies, aiming to enhance the reliability and effectiveness of LLM-based applications in real-world settings.