BEYONDWORDS is All You Need: Agentic Generative AI based Social Media Themes Extractor
This provides a scalable framework for improving thematic analysis in social media, particularly for communities like the autistic one, though it appears incremental in combining existing techniques.
The study tackled thematic analysis of social media posts by introducing a method that integrates tweet embeddings, dimensionality reduction, and generative AI to identify latent themes, applied to tweets from the autistic community to uncover key insights.
Thematic analysis of social media posts provides a major understanding of public discourse, yet traditional methods often struggle to capture the complexity and nuance of unstructured, large-scale text data. This study introduces a novel methodology for thematic analysis that integrates tweet embeddings from pre-trained language models, dimensionality reduction using and matrix factorization, and generative AI to identify and refine latent themes. Our approach clusters compressed tweet representations and employs generative AI to extract and articulate themes through an agentic Chain of Thought (CoT) prompting, with a secondary LLM for quality assurance. This methodology is applied to tweets from the autistic community, a group that increasingly uses social media to discuss their experiences and challenges. By automating the thematic extraction process, the aim is to uncover key insights while maintaining the richness of the original discourse. This autism case study demonstrates the utility of the proposed approach in improving thematic analysis of social media data, offering a scalable and adaptable framework that can be applied to diverse contexts. The results highlight the potential of combining machine learning and Generative AI to enhance the depth and accuracy of theme identification in online communities.