Context-Aware Human Behavior Prediction Using Multimodal Large Language Models: Challenges and Insights
This work addresses the challenge of safe and efficient human-robot interaction through context-aware behavior prediction, but it is incremental as it focuses on benchmarking and modular framework analysis rather than introducing a new paradigm.
The paper tackles the problem of predicting human behavior in shared environments by applying pre-trained Multimodal Large Language Models (MLLMs), achieving 92.8% semantic similarity and 66.1% exact label accuracy in predictions.
Predicting human behavior in shared environments is crucial for safe and efficient human-robot interaction. Traditional data-driven methods to that end are pre-trained on domain-specific datasets, activity types, and prediction horizons. In contrast, the recent breakthroughs in Large Language Models (LLMs) promise open-ended cross-domain generalization to describe various human activities and make predictions in any context. In particular, Multimodal LLMs (MLLMs) are able to integrate information from various sources, achieving more contextual awareness and improved scene understanding. The difficulty in applying general-purpose MLLMs directly for prediction stems from their limited capacity for processing large input sequences, sensitivity to prompt design, and expensive fine-tuning. In this paper, we present a systematic analysis of applying pre-trained MLLMs for context-aware human behavior prediction. To this end, we introduce a modular multimodal human activity prediction framework that allows us to benchmark various MLLMs, input variations, In-Context Learning (ICL), and autoregressive techniques. Our evaluation indicates that the best-performing framework configuration is able to reach 92.8% semantic similarity and 66.1% exact label accuracy in predicting human behaviors in the target frame.