Machine-in-the-Loop Rewriting for Creative Image Captioning
This addresses the challenge of human-AI collaboration in creative writing tasks, offering an incremental improvement over prior methods by enhancing user control and output quality.
The paper tackled the problem of machine-in-the-loop writing by developing a rewriting model that modifies user-specified text spans to add descriptive and figurative elements, enabling better human control in creative image captioning. The result showed that users rated the model as more helpful than a baseline and produced more descriptive and figurative captions when collaborating with it.
Machine-in-the-loop writing aims to enable humans to collaborate with models to complete their writing tasks more effectively. Prior work has found that providing humans a machine-written draft or sentence-level continuations has limited success since the generated text tends to deviate from humans' intention. To allow the user to retain control over the content, we train a rewriting model that, when prompted, modifies specified spans of text within the user's original draft to introduce descriptive and figurative elements locally in the text. We evaluate the model on its ability to collaborate with humans on the task of creative image captioning. On a user study through Amazon Mechanical Turk, our model is rated to be more helpful than a baseline infilling language model. In addition, third-party evaluation shows that users write more descriptive and figurative captions when collaborating with our model compared to completing the task alone.