FaaF: Facts as a Function for the evaluation of generated text
This addresses the need for accurate and efficient verification of information in LM-generated texts, particularly for applications like Retrieval Augmented Generation (RAG) systems, though it appears incremental as it builds on existing LM capabilities.
The paper tackled the problem of unreliable fact verification in text generated by large language models (LMs) by introducing Facts as a Function (FaaF), which significantly enhances the ability to identify unsupported facts while improving efficiency and lowering costs compared to prompt-based methods.
The demand for accurate and efficient verification of information in texts generated by large language models (LMs) is at an all-time high, but remains unresolved. Recent efforts have focused on extracting and verifying atomic facts from these texts via prompting LM evaluators. However, we demonstrate that this method of prompting is unreliable when faced with incomplete or inaccurate reference information. We introduce Facts as a Function (FaaF), a new approach to the fact verification task that leverages the function-calling capabilities of LMs. FaaF significantly enhances the ability of LMs to identify unsupported facts in texts, while also improving efficiency and significantly lowering costs compared to prompt-based methods. Additionally, we propose a framework for evaluating factual recall in Retrieval Augmented Generation (RAG) systems, which we employ to compare prompt-based and FaaF methods using various LMs under challenging conditions.