LLaSA: A Sensor-Aware LLM for Natural Language Reasoning of Human Activity from IMU Data
This addresses the need for explainable activity recognition in wearable systems, though it is incremental as it builds on existing LLM and sensor data methods.
The paper tackled the problem of wearable systems failing to explain the causes or context of human activities from IMU data by introducing two large-scale datasets (SensorCap and OpenSQA) and developing LLaSA, a family of sensor-aware language models that generate interpretable responses, outperforming commercial LLMs like GPT-3.5 and GPT-4o-mini on benchmark and real-world tasks.
Wearable systems can recognize activities from IMU data but often fail to explain their underlying causes or contextual significance. To address this limitation, we introduce two large-scale resources: SensorCap, comprising 35,960 IMU--caption pairs, and OpenSQA, with 199,701 question--answer pairs designed for causal and explanatory reasoning. OpenSQA includes a curated tuning split (Tune-OpenSQA) optimized for scientific accuracy, narrative clarity, and diagnostic insight. Leveraging these datasets, we develop LLaSA (Large Language and Sensor Assistant), a family of compact sensor-aware language models (7B and 13B) that generate interpretable, context-rich responses to open-ended questions grounded in raw IMU data. LLaSA outperforms commercial LLMs, including GPT-3.5 and GPT-4o-mini, on benchmark and real-world tasks, demonstrating the effectiveness of domain supervision and model alignment for sensor reasoning. Our code repository and datasets can be found at https://github.com/BASHLab/LLaSA.