Forecasting With LLMs: Improved Generalization Through Feature Steering
For practitioners using LLMs for forecasting, this work provides a method to improve generalization by steering interpretable temporal features, though the effect is limited to one direction.
The paper investigates whether LLMs rely on time-specific knowledge or generalizable patterns in forecasting tasks, using sparse autoencoders to identify temporal features. By amplifying time-awareness features, they reduce look-ahead bias in forecasting prompts while preserving general reasoning, but steering look-ahead-bias features has no effect.
Successful forecasting involves identifying patterns between historical and future states of the world which generalize to future observations. We apply LLMs to a variety of forecasting tasks and inspect their internal states using sparse autoencoders to understand whether they appear to rely on time-specific pieces of knowledge versus generalizable patterns. Our analyses identify features associated with both time-aware reasoning and look-ahead-biased reasoning. We then apply the LLMs to an entirely different domain and intervene on these features. We find that amplifying time-awareness features substantially reduces look-ahead bias on forecasting prompts while preserving general reasoning performance. In contrast, steering the candidate look-ahead-bias features does not produce an effect. These results suggest that interpretable temporal features can be used to causally shift LLMs toward more historically grounded reasoning.