Measuring Behavior Portability in Large Language Models
For researchers and practitioners evaluating LLM behavior, this work highlights a critical fragility in suite-based evaluation, showing that results are not portable across payoff-equivalent environments.
The paper introduces a formal framework to measure behavioral portability in LLMs, finding substantial and systematic portability losses across seven economic decision problems, indicating that behavioral characterizations from one environment do not reliably transfer to structurally equivalent alternatives.
Large language models are increasingly deployed as autonomous decision makers, yet the behavioral mapping they exhibit can vary substantially across decision environments that are payoff-equivalent by construction-environments that share identical payoff-relevant structure but differ in surface presentation. This sensitivity renders suite-based evaluation fragile and raises a fundamental question of behavioral portability: how well does a behavioral mapping learned in one decision environment informative on another that preserves the same underlying incentive structure? We introduce a formal framework to measure this property. Our protocol fits an interpretable behavioral model on data pooled from a set of source environments and evaluates its out-of-sample predictive performance in a held-out target environment, benchmarking against an oracle trained directly on target data. Portability is quantified via a loss-agnostic measure that delivers worst-case bounds on the performance of the induced prediction-action mapping in the target environment. In controlled experiments spanning seven canonical economic decision problems, we document substantial and systematic portability losses, suggesting that behavioral characterizations of LLMs obtained in one decision environment cannot be assumed to transfer reliably to structurally equivalent alternatives.