Towards Effective Theory of LLMs: A Representation Learning Approach
For researchers seeking to interpret and control LLM behavior, RET provides a method to extract meaningful high-level variables from complex neural computations.
The paper introduces Representational Effective Theory (RET), a framework that learns macrovariables from LLM hidden states to capture high-level structure for interpretability. Results show RET yields temporally consistent states that reveal reasoning trajectories, predict sycophancy early, and enable steering of generations.
We propose Representational Effective Theory (RET), a framework for describing large language model computation in terms of learned macrostates rather than microscopic details. RET learns these macrostates from hidden-state trajectories using a BYOL/JEPA-style self-supervised objective, coarse-graining activations into macrovariables that preserve higher-level structure relevant for prediction and interpretation. We evaluate whether these macrovariables are practically relevant for interpretability: RET yields temporally consistent states that reveal "mental-state" trajectories of reasoning, capture high-level semantic structure, support early prediction of behavioral outcomes such as sycophancy, and provide causal handles for steering generations toward interpretable computational phases. Together, these results suggest that LLM computation admits useful effective descriptions via RET: high-level, dynamically meaningful variables that support interpretation, prediction, and intervention.