Towards AI Evaluation in Domain-Specific RAG Systems: The AgriHubi Case Study
This work addresses the challenge of providing agricultural decision support in low-resource languages, though it is incremental as it adapts existing RAG methods to a specific domain and language.
The paper tackled the problem of using large language models in agriculture, particularly for low-resource languages like Finnish, by developing AgriHubi, a domain-adapted RAG system that showed gains in answer completeness, linguistic accuracy, and perceived reliability through user studies.
Large language models show promise for knowledge-intensive domains, yet their use in agriculture is constrained by weak grounding, English-centric training data, and limited real-world evaluation. These issues are amplified for low-resource languages, where high-quality domain documentation exists but remains difficult to access through general-purpose models. This paper presents AgriHubi, a domain-adapted retrieval-augmented generation (RAG) system for Finnish-language agricultural decision support. AgriHubi integrates Finnish agricultural documents with open PORO family models and combines explicit source grounding with user feedback to support iterative refinement. Developed over eight iterations and evaluated through two user studies, the system shows clear gains in answer completeness, linguistic accuracy, and perceived reliability. The results also reveal practical trade-offs between response quality and latency when deploying larger models. This study provides empirical guidance for designing and evaluating domain-specific RAG systems in low-resource language settings.