iCap: Interactive Image Captioning with Predictive Text
This addresses the problem of enhancing human-AI collaboration in image captioning for users, though it is incremental as it builds on existing captioning methods.
The paper tackles interactive image captioning by introducing Visually Conditioned Sentence Completion (VCSC) and building the iCap system, which predicts text from live user input and shows viability through experiments and user studies.
In this paper we study a brand new topic of interactive image captioning with human in the loop. Different from automated image captioning where a given test image is the sole input in the inference stage, we have access to both the test image and a sequence of (incomplete) user-input sentences in the interactive scenario. We formulate the problem as Visually Conditioned Sentence Completion (VCSC). For VCSC, we propose asynchronous bidirectional decoding for image caption completion (ABD-Cap). With ABD-Cap as the core module, we build iCap, a web-based interactive image captioning system capable of predicting new text with respect to live input from a user. A number of experiments covering both automated evaluations and real user studies show the viability of our proposals.