Towards Requirements Engineering for RAG Systems
This work addresses requirements engineering for RAG systems in complex domain-specific applications like maritime services, but it is incremental, focusing on empirical insights rather than broad solutions.
The paper tackles the challenge of aligning user expectations with output correctness in Retrieval Augmented Generation (RAG) systems for expert domains, finding that data scientists must iteratively identify context-specific retrieval requirements through collaboration with users.
This short paper explores how a maritime company develops and integrates large-language models (LLM). Specifically by looking at the requirements engineering for Retrieval Augmented Generation (RAG) systems in expert settings. Through a case study at a maritime service provider, we demonstrate how data scientists face a fundamental tension between user expectations of AI perfection and the correctness of the generated outputs. Our findings reveal that data scientists must identify context-specific "retrieval requirements" through iterative experimentation together with users because they are the ones who can determine correctness. We present an empirical process model describing how data scientists practically elicited these "retrieval requirements" and managed system limitations. This work advances software engineering knowledge by providing insights into the specialized requirements engineering processes for implementing RAG systems in complex domain-specific applications.