Large Language Models can Share Images, Too!
It addresses the need for assessing LLMs' emergent abilities in image-sharing for applications like human-bot interaction and dataset augmentation, but is incremental as it builds on existing LLM capabilities.
This paper tackles the problem of evaluating and enabling image-sharing capabilities in Large Language Models (LLMs) like GPT-4 and LLaMA 2 in a zero-shot setting, introducing the DribeR framework and PhotoChat++ dataset, with ChatGPT achieving the best performance in experiments.
This paper explores the image-sharing capability of Large Language Models (LLMs), such as GPT-4 and LLaMA 2, in a zero-shot setting. To facilitate a comprehensive evaluation of LLMs, we introduce the PhotoChat++ dataset, which includes enriched annotations (i.e., intent, triggering sentence, image description, and salient information). Furthermore, we present the gradient-free and extensible Decide, Describe, and Retrieve (DribeR) framework. With extensive experiments, we unlock the image-sharing capability of DribeR equipped with LLMs in zero-shot prompting, with ChatGPT achieving the best performance. Our findings also reveal the emergent image-sharing ability in LLMs under zero-shot conditions, validating the effectiveness of DribeR. We use this framework to demonstrate its practicality and effectiveness in two real-world scenarios: (1) human-bot interaction and (2) dataset augmentation. To the best of our knowledge, this is the first study to assess the image-sharing ability of various LLMs in a zero-shot setting. We make our source code and dataset publicly available at https://github.com/passing2961/DribeR.