Polar probe linearly decodes semantic structures from LLMs
Provides a simple, interpretable mechanism for how LLMs bind concepts into complex semantic structures, offering insights for model analysis and improvement.
The paper proposes that LLMs represent semantic structures via a geometric code where distance and direction between entity embeddings encode relation existence and type. Polar probes linearly decode these structures across five domains, with decoding quality correlating with LLM performance.
How do artificial neural networks bind concepts to form complex semantic structures? Here, we propose a simple neural code, whereby the existence and the type of relations between entities are represented by the distance and the direction between their embeddings, respectively. We test this hypothesis in a variety of Large Language Models (LLMs), each input with natural-language descriptions of minimalist tasks from five different domains: arithmetic, visual scenes, family trees, metro maps and social interactions. Results show that the true semantic structures can be linearly recovered with a Polar Probe targeting a subspace of LLMs' layer activations. Second, this code emerges mostly in middle layers and improves with LLM performance. Third, these Polar Probes successfully generalize to new entities and relation types, but degrades with the size of the semantic structure. Finally, the quality of the polar representation correlates with the LLM's ability to answer questions about the semantic structure. Together, these findings suggest that LLMs learn to build complex semantic structures by binding representations with a simple geometrical principle.