Exploring Social Media Image Categorization Using Large Models with Different Adaptation Methods: A Case Study on Cultural Nature's Contributions to People
This work tackles the problem of categorizing complex social media images for researchers and practitioners interested in human-nature interactions and cultural heritage, offering a new dataset and a comparative analysis of large model solutions.
This paper addresses the challenge of categorizing social media images, particularly those depicting human-nature interactions with abstract concepts, by introducing the FLIPS dataset. It evaluates various large models (LLMs, LVMs, LVLMs) and adaptation methods, reporting their performance across cost, productivity, scalability, and result quality.
Social media images provide valuable insights for modeling, mapping, and understanding human interactions with natural and cultural heritage. However, categorizing these images into semantically meaningful groups remains highly complex due to the vast diversity and heterogeneity of their visual content as they contain an open-world human and nature elements. This challenge becomes greater when categories involve abstract concepts and lack consistent visual patterns. Related studies involve human supervision in the categorization process and the lack of public benchmark datasets make comparisons between these works unfeasible. On the other hand, the continuous advances in large models, including Large Language Models (LLMs), Large Visual Models (LVMs), and Large Visual Language Models (LVLMs), provide a large space of unexplored solutions. In this work 1) we introduce FLIPS a dataset of Flickr images that capture the interaction between human and nature, and 2) evaluate various solutions based on different types and combinations of large models using various adaptation methods. We assess and report their performance in terms of cost, productivity, scalability, and result quality to address the challenges of social media image categorization.