Sample Efficient Multimodal Semantic Augmentation for Incremental Summarization
This work addresses video summarization for task-oriented applications, but appears incremental as it extends existing methods to videos with clustering/querying modifications.
The paper tackles incremental summarization of task videos by developing a sample-efficient few-shot prompting approach that extracts semantic concepts as an intermediate step, showing that richer input context with relevant entities and actions enhances generated summaries.
In this work, we develop a prompting approach for incremental summarization of task videos. We develop a sample-efficient few-shot approach for extracting semantic concepts as an intermediate step. We leverage an existing model for extracting the concepts from the images and extend it to videos and introduce a clustering and querying approach for sample efficiency, motivated by the recent advances in perceiver-based architectures. Our work provides further evidence that an approach with richer input context with relevant entities and actions from the videos and using these as prompts could enhance the summaries generated by the model. We show the results on a relevant dataset and discuss possible directions for the work.