Towards Consistent Language Models Using Declarative Constraints
This addresses the issue of unreliable AI responses for users needing accurate information, but it is incremental as it builds on existing data management methods.
The paper tackles the problem of language models generating inconsistent and incorrect answers by proposing to use declarative constraints from data management to modify them, reporting preliminary empirical studies on achieving consistent and relevant outputs.
Large language models have shown unprecedented abilities in generating linguistically coherent and syntactically correct natural language output. However, they often return incorrect and inconsistent answers to input questions. Due to the complexity and uninterpretability of the internally learned representations, it is challenging to modify language models such that they provide correct and consistent results. The data management community has developed various methods and tools for providing consistent answers over inconsistent datasets. In these methods, users specify the desired properties of data in a domain in the form of high-level declarative constraints. This approach has provided usable and scalable methods to delivering consistent information from inconsistent datasets. We aim to build upon this success and leverage these methods to modify language models such that they deliver consistent and accurate results. We investigate the challenges of using these ideas to obtain consistent and relevant answers from language models and report some preliminary empirical studies.