Beyond the Surface: Probing the Ideological Depth of Large Language Models
This work addresses the issue of understanding political biases and capabilities in LLMs for researchers and developers, but it is incremental as it builds on existing probing and steering methods.
The paper tackled the problem of measuring ideological depth in large language models by defining it as steerability and internal political feature richness, finding that Gemma-2-9B-IT is more steerable and has 7.3x more distinct political features than Llama-3.1-8B-Instruct, with ablations showing that feature deficits increase refusal rates.
Large language models (LLMs) display recognizable political leanings, yet they vary significantly in their ability to represent a political orientation consistently. In this paper, we define ideological depth as (i) a model's ability to follow political instructions without failure (steerability), and (ii) the feature richness of its internal political representations measured with sparse autoencoders (SAEs), an unsupervised sparse dictionary learning (SDL) approach. Using Llama-3.1-8B-Instruct and Gemma-2-9B-IT as candidates, we compare prompt-based and activation-steering interventions and probe political features with publicly available SAEs. We find large, systematic differences: Gemma is more steerable in both directions and activates approximately 7.3x more distinct political features than Llama. Furthermore, causal ablations of a small targeted set of Gemma's political features to create a similar feature-poor setting induce consistent shifts in its behavior, with increased rates of refusals across topics. Together, these results indicate that refusals on benign political instructions or prompts can arise from capability deficits rather than safety guardrails. Ideological depth thus emerges as a measurable property of LLMs, and steerability serves as a window into their latent political architecture.