MobQA: A Benchmark Dataset for Semantic Understanding of Human Mobility Data through Question Answering
This provides a new benchmark for assessing semantic understanding in mobility data, which is important for researchers in AI and human-computer interaction, though it is incremental as it builds on existing evaluation frameworks.
The paper tackles the problem of evaluating how well large language models (LLMs) understand the semantic meaning of human mobility data, by introducing MobQA, a benchmark dataset with 5,800 question-answer pairs, and finds that LLMs perform well on factual retrieval but have significant limitations in semantic reasoning and explanation tasks.
This paper presents MobQA, a benchmark dataset designed to evaluate the semantic understanding capabilities of large language models (LLMs) for human mobility data through natural language question answering. While existing models excel at predicting human movement patterns, it remains unobvious how much they can interpret the underlying reasons or semantic meaning of those patterns. MobQA provides a comprehensive evaluation framework for LLMs to answer questions about diverse human GPS trajectories spanning daily to weekly granularities. It comprises 5,800 high-quality question-answer pairs across three complementary question types: factual retrieval (precise data extraction), multiple-choice reasoning (semantic inference), and free-form explanation (interpretive description), which all require spatial, temporal, and semantic reasoning. Our evaluation of major LLMs reveals strong performance on factual retrieval but significant limitations in semantic reasoning and explanation question answering, with trajectory length substantially impacting model effectiveness. These findings demonstrate the achievements and limitations of state-of-the-art LLMs for semantic mobility understanding.\footnote{MobQA dataset is available at https://github.com/CyberAgentAILab/mobqa.}