Soft Measures for Extracting Causal Collective Intelligence
This work addresses the problem of automating causal mental model extraction for researchers in social systems and NLP, but it is incremental as it highlights existing measures' shortcomings.
The study tackled the challenge of automatically extracting fuzzy cognitive maps (FCMs) from text to model collective intelligence, using large language models and novel graph-based similarity measures, with results showing positive correlations with human judgments but limitations in capturing FCM nuances.
Understanding and modeling collective intelligence is essential for addressing complex social systems. Directed graphs called fuzzy cognitive maps (FCMs) offer a powerful tool for encoding causal mental models, but extracting high-integrity FCMs from text is challenging. This study presents an approach using large language models (LLMs) to automate FCM extraction. We introduce novel graph-based similarity measures and evaluate them by correlating their outputs with human judgments through the Elo rating system. Results show positive correlations with human evaluations, but even the best-performing measure exhibits limitations in capturing FCM nuances. Fine-tuning LLMs improves performance, but existing measures still fall short. This study highlights the need for soft similarity measures tailored to FCM extraction, advancing collective intelligence modeling with NLP.