Grounded Intuition of GPT-Vision's Abilities with Scientific Images
This work helps researchers develop grounded intuitions for new models, particularly in making scientific information more accessible, but it is incremental as it adapts existing qualitative methods to NLP.
The study tackled the challenge of understanding GPT-Vision's capabilities and limitations by developing a qualitative evaluation framework based on grounded theory and thematic analysis, finding that GPT-Vision is sensitive to prompting, counterfactual text, and spatial relationships in alt text generation for scientific figures.
GPT-Vision has impressed us on a range of vision-language tasks, but it comes with the familiar new challenge: we have little idea of its capabilities and limitations. In our study, we formalize a process that many have instinctively been trying already to develop "grounded intuition" of this new model. Inspired by the recent movement away from benchmarking in favor of example-driven qualitative evaluation, we draw upon grounded theory and thematic analysis in social science and human-computer interaction to establish a rigorous framework for qualitative evaluation in natural language processing. We use our technique to examine alt text generation for scientific figures, finding that GPT-Vision is particularly sensitive to prompting, counterfactual text in images, and relative spatial relationships. Our method and analysis aim to help researchers ramp up their own grounded intuitions of new models while exposing how GPT-Vision can be applied to make information more accessible.