Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognition
This addresses the challenge of efficiently processing high-frequency sensor data for human activity recognition in IoT applications, though it appears incremental as it adapts existing visual prompting methods to a specific domain.
The paper tackles the problem of information loss and high token costs when applying large language models to high-frequency eye-tracking data for human activity recognition by investigating visual prompting strategies that transform sensor signals into visualization images. The results show that visual prompting provides a token-efficient and scalable representation, enabling multimodal LLMs to effectively reason over eye-tracking data across three public datasets.
Large Language Models (LLMs) have emerged as foundation models for IoT applications such as human activity recognition (HAR). However, directly applying high-frequency and multi-dimensional sensor data, such as eye-tracking data, leads to information loss and high token costs. To mitigate this, we investigate a visual prompting strategy that transforms sensor signals into data visualization images as an input to multimodal LLMs (MLLMs) using eye-tracking data. We conducted a systematic evaluation of MLLM-based HAR across three public eye-tracking datasets using three visualization types of timeline, heatmap, and scanpath, under varying temporal window sizes. Our findings suggest that visual prompting provides a token-efficient and scalable representation for eye-tracking data, highlighting its potential to enable MLLMs to effectively reason over high-frequency sensor signals in IoT contexts.