Conveying Meaning through Gestures: An Investigation into Semantic Co-Speech Gesture Generation
This research addresses the problem of generating meaningful gestures for human-computer interaction, but it is incremental as it builds on existing frameworks to explore semantic augmentation.
The study investigated two co-speech gesture generation frameworks, AQ-GT and AQ-GT-a, to assess their ability to convey meaning through gestures and human perception. Results showed that AQ-GT was more effective at conveying concepts within its training domain, while AQ-GT-a demonstrated better generalization for shape and size in novel contexts, though it was not perceived as more human-like.
This study explores two frameworks for co-speech gesture generation, AQ-GT and its semantically-augmented variant AQ-GT-a, to evaluate their ability to convey meaning through gestures and how humans perceive the resulting movements. Using sentences from the SAGA spatial communication corpus, contextually similar sentences, and novel movement-focused sentences, we conducted a user-centered evaluation of concept recognition and human-likeness. Results revealed a nuanced relationship between semantic annotations and performance. The original AQ-GT framework, lacking explicit semantic input, was surprisingly more effective at conveying concepts within its training domain. Conversely, the AQ-GT-a framework demonstrated better generalization, particularly for representing shape and size in novel contexts. While participants rated gestures from AQ-GT-a as more expressive and helpful, they did not perceive them as more human-like. These findings suggest that explicit semantic enrichment does not guarantee improved gesture generation and that its effectiveness is highly dependent on the context, indicating a potential trade-off between specialization and generalization.