Are LLMs Correctly Integrated into Software Systems?
This addresses challenges for developers in integrating LLMs into software systems, but it is incremental as it builds on existing RAG methods to identify and resolve defects.
The paper tackled the problem of integrating large language models (LLMs) with retrieval-augmented generation (RAG) into software systems by studying 100 open-source applications, identifying 18 defect patterns and finding that 77% of applications had more than three types of integration defects that degraded functionality, efficiency, and security.
Large language models (LLMs) provide effective solutions in various application scenarios, with the support of retrieval-augmented generation (RAG). However, developers face challenges in integrating LLM and RAG into software systems, due to lacking interface specifications, various requirements from software context, and complicated system management. In this paper, we have conducted a comprehensive study of 100 open-source applications that incorporate LLMs with RAG support, and identified 18 defect patterns. Our study reveals that 77% of these applications contain more than three types of integration defects that degrade software functionality, efficiency, and security. Guided by our study, we propose systematic guidelines for resolving these defects in software life cycle. We also construct an open-source defect library Hydrangea.