Challenges with unsupervised LLM knowledge discovery
This work highlights a critical flaw in unsupervised knowledge elicitation for LLMs, which is important for researchers in AI interpretability and safety, though it is incremental in exposing limitations rather than providing a new solution.
The paper demonstrates that existing unsupervised methods for discovering knowledge in large language model activations fail to actually identify knowledge, instead detecting the most prominent activation features, as proven theoretically and shown experimentally. It concludes these methods are insufficient and proposes sanity checks for future evaluations.
We show that existing unsupervised methods on large language model (LLM) activations do not discover knowledge -- instead they seem to discover whatever feature of the activations is most prominent. The idea behind unsupervised knowledge elicitation is that knowledge satisfies a consistency structure, which can be used to discover knowledge. We first prove theoretically that arbitrary features (not just knowledge) satisfy the consistency structure of a particular leading unsupervised knowledge-elicitation method, contrast-consistent search (Burns et al. - arXiv:2212.03827). We then present a series of experiments showing settings in which unsupervised methods result in classifiers that do not predict knowledge, but instead predict a different prominent feature. We conclude that existing unsupervised methods for discovering latent knowledge are insufficient, and we contribute sanity checks to apply to evaluating future knowledge elicitation methods. Conceptually, we hypothesise that the identification issues explored here, e.g. distinguishing a model's knowledge from that of a simulated character's, will persist for future unsupervised methods.