Seeing the Goal, Missing the Truth: Human Accountability for AI Bias
This addresses AI bias issues for researchers and practitioners by highlighting human accountability in research design, though it is incremental in linking goal leakage to bias.
The study investigated how human-defined goals cause bias in Large Language Models (LLMs) during financial prediction tasks, finding that revealing downstream uses leads to biased sentiment and competition measures, with performance improvements only before the model's knowledge cutoff.
This research explores how human-defined goals influence the behavior of Large Language Models (LLMs) through purpose-conditioned cognition. Using financial prediction tasks, we show that revealing the downstream use (e.g., predicting stock returns or earnings) of LLM outputs leads the LLM to generate biased sentiment and competition measures, even though these measures are intended to be downstream task-independent. Goal-aware prompting shifts intermediate measures toward the disclosed downstream objective. This purpose leakage improves performance before the LLM's knowledge cutoff, but with no advantage post-cutoff. AI bias due to "seeing the goal" is not an algorithmic flaw, but stems from human accountability in research design to ensure the statistical validity and reliability of AI-generated measurements.