Probing Language Models' Gesture Understanding for Enhanced Human-AI Interaction
This work addresses the challenge of enhancing human-AI interaction by exploring LLMs' gesture comprehension, but it is incremental as it applies existing psycholinguistic methods to a new domain without claiming breakthroughs.
The researchers tackled the problem of assessing large language models' (LLMs) understanding of gestures in human-AI interaction by proposing to test their proficiency in deciphering explicit and implicit non-verbal cues from textual prompts, with results planned to measure agreement between LLM-identified gestures and a constructed dataset.
The rise of Large Language Models (LLMs) has affected various disciplines that got beyond mere text generation. Going beyond their textual nature, this project proposal aims to investigate the interaction between LLMs and non-verbal communication, specifically focusing on gestures. The proposal sets out a plan to examine the proficiency of LLMs in deciphering both explicit and implicit non-verbal cues within textual prompts and their ability to associate these gestures with various contextual factors. The research proposes to test established psycholinguistic study designs to construct a comprehensive dataset that pairs textual prompts with detailed gesture descriptions, encompassing diverse regional variations, and semantic labels. To assess LLMs' comprehension of gestures, experiments are planned, evaluating their ability to simulate human behaviour in order to replicate psycholinguistic experiments. These experiments consider cultural dimensions and measure the agreement between LLM-identified gestures and the dataset, shedding light on the models' contextual interpretation of non-verbal cues (e.g. gestures).