Explicit Modelling of Theory of Mind for Belief Prediction in Nonverbal Social Interactions
This work addresses the challenge of enabling artificial systems to better collaborate with humans by robustly predicting beliefs from nonverbal behavior, representing a domain-specific advancement in social AI.
The authors tackled the problem of predicting human beliefs and their dynamics from nonverbal social interactions by proposing MToMnet, a neural network that explicitly models Theory of Mind using multimodal inputs, and demonstrated that it surpasses existing methods by a large margin while requiring significantly fewer parameters.
We propose MToMnet - a Theory of Mind (ToM) neural network for predicting beliefs and their dynamics during human social interactions from multimodal input. ToM is key for effective nonverbal human communication and collaboration, yet, existing methods for belief modelling have not included explicit ToM modelling or have typically been limited to one or two modalities. MToMnet encodes contextual cues (scene videos and object locations) and integrates them with person-specific cues (human gaze and body language) in a separate MindNet for each person. Inspired by prior research on social cognition and computational ToM, we propose three different MToMnet variants: two involving fusion of latent representations and one involving re-ranking of classification scores. We evaluate our approach on two challenging real-world datasets, one focusing on belief prediction, while the other examining belief dynamics prediction. Our results demonstrate that MToMnet surpasses existing methods by a large margin while at the same time requiring a significantly smaller number of parameters. Taken together, our method opens up a highly promising direction for future work on artificial intelligent systems that can robustly predict human beliefs from their non-verbal behaviour and, as such, more effectively collaborate with humans.