Multimodal Cognitive Reframing Therapy via Multi-hop Psychotherapeutic Reasoning
This work addresses the problem of enhancing AI-assisted psychotherapy for virtual clients by incorporating non-verbal evidence, though it is incremental by building on existing LLM-based methods.
The paper tackles the limitation of text-only cognitive reframing therapy by extending it to multimodality with visual clues, resulting in improved performance of vision-language models as psychotherapists and more thoughtful suggestions using a multi-hop reasoning approach.
Previous research has revealed the potential of large language models (LLMs) to support cognitive reframing therapy; however, their focus was primarily on text-based methods, often overlooking the importance of non-verbal evidence crucial in real-life therapy. To alleviate this gap, we extend the textual cognitive reframing to multimodality, incorporating visual clues. Specifically, we present a new dataset called Multi Modal-Cognitive Support Conversation (M2CoSC), which pairs each GPT-4-generated dialogue with an image that reflects the virtual client's facial expressions. To better mirror real psychotherapy, where facial expressions lead to interpreting implicit emotional evidence, we propose a multi-hop psychotherapeutic reasoning approach that explicitly identifies and incorporates subtle evidence. Our comprehensive experiments with both LLMs and vision-language models (VLMs) demonstrate that the VLMs' performance as psychotherapists is significantly improved with the M2CoSC dataset. Furthermore, the multi-hop psychotherapeutic reasoning method enables VLMs to provide more thoughtful and empathetic suggestions, outperforming standard prompting methods.