What are They Thinking? Delineation, Probing and Tracking of Concepts in LLMs
For AI safety researchers and practitioners, this work provides a foundational step toward scalable concept monitoring in LLMs, though it is incremental as it focuses on methodology rather than achieving broad SOTA.
This paper introduces a method for creating low-cost linear probes to detect concepts in LLM embeddings, demonstrating that probes can track concepts across contexts with four concepts and three models, aiming to enable scalable monitoring of model reasoning.
As the influence of LLMs expands, it is imperative to gain insight into their decisions. One way to do that is to develop probes that detect the presence or absence of a broad set of concepts within the embeddings computed in an LLM - which is what we might say a model is "thinking" about. Such probes should be low-cost and easily applicable to any LLM, so that monitoring for many concepts is possible during normal operation. In this paper, we take the first steps towards developing the capability of creating many such probes by defining and executing examples of the key tasks needed: first, the careful delineation of a concept through the creation of a dataset with the concept both present and then absent. Then, the training and testing of a set of linear probes to detect the concept on any layer of an LLM, including an exploration of the complexity of the probe needed. Finally, we show that such probes can track concepts across larger contexts. This is done with four separate concepts and three different LLMs. When this process is scaled to many more concepts, it will create the ability to easily monitor new models.