Figuring out Figures: Using Textual References to Caption Scientific Figures
This work addresses the challenge of improving caption generation for scientific figures, which is important for researchers and automated systems, but it is incremental as it builds on existing datasets and models.
The paper tackles the problem of automatically generating captions for scientific figures by using textual references from the paper, such as titles and abstracts, alongside image data, and found that a model using only textual metadata achieved optimal performance.
Figures are essential channels for densely communicating complex ideas in scientific papers. Previous work in automatically generating figure captions has been largely unsuccessful and has defaulted to using single-layer LSTMs, which no longer achieve state-of-the-art performance. In our work, we use the SciCap datasets curated by Hsu et al. and use a variant of a CLIP+GPT-2 encoder-decoder model with cross-attention to generate captions conditioned on the image. Furthermore, we augment our training pipeline by creating a new dataset MetaSciCap that incorporates textual metadata from the original paper relevant to the figure, such as the title, abstract, and in-text references. We use SciBERT to encode the textual metadata and use this encoding alongside the figure embedding. In our experimentation with different models, we found that the CLIP+GPT-2 model performs better when it receives all textual metadata from the SciBERT encoder in addition to the figure, but employing a SciBERT+GPT2 model that uses only the textual metadata achieved optimal performance.