ST-LINK: Spatially-Aware Large Language Models for Spatio-Temporal Forecasting
This work solves the challenge of integrating spatial awareness into LLMs for spatio-temporal forecasting, particularly in intelligent transportation systems, representing an incremental improvement over existing methods.
The paper tackles the problem of traffic forecasting by addressing the limitations of Large Language Models (LLMs) in capturing spatial dependencies, introducing ST-LINK with Spatially-Enhanced Attention and Memory Retrieval Feed-Forward Network, which outperforms conventional deep learning and LLM approaches on benchmark datasets.
Traffic forecasting represents a crucial problem within intelligent transportation systems. In recent research, Large Language Models (LLMs) have emerged as a promising method, but their intrinsic design, tailored primarily for sequential token processing, introduces notable challenges in effectively capturing spatial dependencies. Specifically, the inherent limitations of LLMs in modeling spatial relationships and their architectural incompatibility with graph-structured spatial data remain largely unaddressed. To overcome these limitations, we introduce ST-LINK, a novel framework that enhances the capability of Large Language Models to capture spatio-temporal dependencies. Its key components are Spatially-Enhanced Attention (SE-Attention) and the Memory Retrieval Feed-Forward Network (MRFFN). SE-Attention extends rotary position embeddings to integrate spatial correlations as direct rotational transformations within the attention mechanism. This approach maximizes spatial learning while preserving the LLM's inherent sequential processing structure. Meanwhile, MRFFN dynamically retrieves and utilizes key historical patterns to capture complex temporal dependencies and improve the stability of long-term forecasting. Comprehensive experiments on benchmark datasets demonstrate that ST-LINK surpasses conventional deep learning and LLM approaches, and effectively captures both regular traffic patterns and abrupt changes.