Interpreting Multi-Attribute Confounding through Numerical Attributes in Large Language Models
This work addresses a vulnerability in LLM decision-making for AI fairness and control, though it is incremental in nature.
The study investigated how large language models (LLMs) internally represent and integrate multiple numerical attributes, finding that they encode real-world correlations but systematically amplify them, with irrelevant context causing shifts in magnitude representations that affect downstream outputs.
Although behavioral studies have documented numerical reasoning errors in large language models (LLMs), the underlying representational mechanisms remain unclear. We hypothesize that numerical attributes occupy shared latent subspaces and investigate two questions:(1) How do LLMs internally integrate multiple numerical attributes of a single entity? (2)How does irrelevant numerical context perturb these representations and their downstream outputs? To address these questions, we combine linear probing with partial correlation analysis and prompt-based vulnerability tests across models of varying sizes. Our results show that LLMs encode real-world numerical correlations but tend to systematically amplify them. Moreover, irrelevant context induces consistent shifts in magnitude representations, with downstream effects that vary by model size. These findings reveal a vulnerability in LLM decision-making and lay the groundwork for fairer, representation-aware control under multi-attribute entanglement.