Imagine How To Change: Explicit Procedure Modeling for Change Captioning
This work addresses the problem of generating more comprehensive descriptions of visual changes, including the 'how', which is important for applications requiring detailed temporal understanding of visual events.
This paper introduces ProCap, a new framework for change captioning that models the temporal dynamics of changes between two images, rather than just comparing static pairs. ProCap uses a two-stage design: first, a procedure encoder learns change dynamics from keyframes, and then this encoder is integrated into an encoder-decoder model using learnable procedure queries for caption generation. Experiments on three datasets demonstrate its effectiveness.
Change captioning generates descriptions that explicitly describe the differences between two visually similar images. Existing methods operate on static image pairs, thus ignoring the rich temporal dynamics of the change procedure, which is the key to understand not only what has changed but also how it occurs. We introduce ProCap, a novel framework that reformulates change modeling from static image comparison to dynamic procedure modeling. ProCap features a two-stage design: The first stage trains a procedure encoder to learn the change procedure from a sparse set of keyframes. These keyframes are obtained by automatically generating intermediate frames to make the implicit procedural dynamics explicit and then sampling them to mitigate redundancy. Then the encoder learns to capture the latent dynamics of these keyframes via a caption-conditioned, masked reconstruction task. The second stage integrates this trained encoder within an encoder-decoder model for captioning. Instead of relying on explicit frames from the previous stage -- a process incurring computational overhead and sensitivity to visual noise -- we introduce learnable procedure queries to prompt the encoder for inferring the latent procedure representation, which the decoder then translates into text. The entire model is then trained end-to-end with a captioning loss, ensuring the encoder's output is both temporally coherent and captioning-aligned. Experiments on three datasets demonstrate the effectiveness of ProCap. Code and pre-trained models are available at https://github.com/BlueberryOreo/ProCap