MuLan: A Study of Fact Mutability in Language Models
This work addresses the challenge of ensuring trustworthy language models for applications requiring accurate time-contingent knowledge, but it is incremental as it builds on existing research in fact mutability and model evaluation.
The study tackled the problem of how language models handle mutable facts that change over time, such as political positions or sports champions, by creating the MuLan benchmark and evaluating six models. The results showed consistent differences in model confidence, representations, and update behavior based on fact mutability, with models encoding mutable facts differently, making them easier to update.
Facts are subject to contingencies and can be true or false in different circumstances. One such contingency is time, wherein some facts mutate over a given period, e.g., the president of a country or the winner of a championship. Trustworthy language models ideally identify mutable facts as such and process them accordingly. We create MuLan, a benchmark for evaluating the ability of English language models to anticipate time-contingency, covering both 1:1 and 1:N relations. We hypothesize that mutable facts are encoded differently than immutable ones, hence being easier to update. In a detailed evaluation of six popular large language models, we consistently find differences in the LLMs' confidence, representations, and update behavior, depending on the mutability of a fact. Our findings should inform future work on the injection of and induction of time-contingent knowledge to/from LLMs.