Multimodal Story Generation on Plural Images
This is an incremental undergraduate project report focused on multimodal story generation for specific applications.
The authors tackled the problem of generating text from multiple images by proposing StoryGen, an architecture that uses images as input for text generation, and demonstrated its ability to produce meaningful paragraphs based on extracted image features.
Traditionally, text generation models take in a sequence of text as input, and iteratively generate the next most probable word using pre-trained parameters. In this work, we propose the architecture to use images instead of text as the input of the text generation model, called StoryGen. In the architecture, we design a Relational Text Data Generator algorithm that relates different features from multiple images. The output samples from the model demonstrate the ability to generate meaningful paragraphs of text containing the extracted features from the input images. This is an undergraduate project report. Completed Dec. 2019 at the Cooper Union.