Testing Conviction: An Argumentative Framework for Measuring LLM Political Stability
This addresses the challenge of measuring political stability in LLMs for researchers and policymakers concerned about AI's influence on discourse, though it is incremental in building on existing classification methods.
The researchers tackled the problem of distinguishing genuine ideological alignment from superficial mimicry in LLMs' political responses by developing a framework that evaluates argumentative consistency and uncertainty quantification. They found that 95% of left-leaning and 89% of right-leaning models demonstrated stable ideological positioning across conditions, with semantic entropy validating classifications (AUROC=0.78).
Large Language Models (LLMs) increasingly shape political discourse, yet exhibit inconsistent responses when challenged. While prior research categorizes LLMs as left- or right-leaning based on single-prompt responses, a critical question remains: Do these classifications reflect stable ideologies or superficial mimicry? Existing methods cannot distinguish between genuine ideological alignment and performative text generation. To address this, we propose a framework for evaluating ideological depth through (1) argumentative consistency and (2) uncertainty quantification. Testing 12 LLMs on 19 economic policies from the Political Compass Test, we classify responses as stable or performative ideological positioning. Results show 95% of left-leaning models and 89% of right-leaning models demonstrate behavior consistent with our classifications across different experimental conditions. Furthermore, semantic entropy strongly validates our classifications (AUROC=0.78), revealing uncertainty's relationship to ideological consistency. Our findings demonstrate that ideological stability is topic-dependent and challenge the notion of monolithic LLM ideologies, and offer a robust way to distinguish genuine alignment from performative behavior.