SensorLM: Learning the Language of Wearable Sensors
This work addresses the problem of interpreting pervasive wearable sensor data for applications like human activity analysis and healthcare, representing a novel method for a known bottleneck.
The paper tackles the challenge of aligning and interpreting wearable sensor data with natural language by introducing SensorLM, a family of sensor-language foundation models, which achieved superior performance in zero-shot recognition, few-shot learning, and cross-modal retrieval on real-world tasks.
We present SensorLM, a family of sensor-language foundation models that enable wearable sensor data understanding with natural language. Despite its pervasive nature, aligning and interpreting sensor data with language remains challenging due to the lack of paired, richly annotated sensor-text descriptions in uncurated, real-world wearable data. We introduce a hierarchical caption generation pipeline designed to capture statistical, structural, and semantic information from sensor data. This approach enabled the curation of the largest sensor-language dataset to date, comprising over 59.7 million hours of data from more than 103,000 people. Furthermore, SensorLM extends prominent multimodal pretraining architectures (e.g., CLIP, CoCa) and recovers them as specific variants within a generic architecture. Extensive experiments on real-world tasks in human activity analysis and healthcare verify the superior performance of SensorLM over state-of-the-art in zero-shot recognition, few-shot learning, and cross-modal retrieval. SensorLM also demonstrates intriguing capabilities including scaling behaviors, label efficiency, sensor captioning, and zero-shot generalization to unseen tasks.