LLM Pretraining Shapes a Generalizable Manifold: Insights into Cross-Modal Transfer to Time Series
For researchers in time-series forecasting and transfer learning, this provides a geometric explanation for why LLMs transfer to time series, but the findings are incremental as they extend known manifold-based transfer concepts to a new modality.
The paper shows that language-pretrained transformers become effective time-series forecasters because pretraining builds a reusable manifold. A linear probe on frozen LLM states decodes realistic trajectories without paired supervision, and finetuning aligns this manifold to task-relevant directions, improving optimization and producing competitive forecasts.
Can language-pretrained transformers become effective time-series forecasters, and why? In this paper, we show that cross-modal transfer arises because language pretraining preconditions time series training with a reusable manifold. A linear probe on frozen LLM states decodes realistic time-series trajectories without paired supervision, and retrieval in this projected space yields competitive forecasts, showing that structure and dynamics exist before finetuning. Pretrained initialization also improves optimization, producing coherent gradients and a highly anisotropic loss landscape unlike random initialization. Finetuning then acts as low-dimensional alignment, reusing existing directions rather than learning temporal primitives from scratch, as evidenced by low-rank updates, subspace alignment, and shared features for periodicity, trend, and repetition. Together, these results support a geometric account of LLM-to-time-series transfer: language pretraining builds the manifold, and finetuning projects numerical dynamics onto task-relevant directions.