Reasoning-Enhanced Object-Centric Learning for Videos
This work addresses the need for better reasoning in video object-centric models, which is crucial for applications in robotics and autonomous systems, though it appears incremental by building on existing slot-based methods.
The paper tackles the problem of object-centric learning in videos by introducing a reasoning module called STATM, which enhances perception in complex scenes through slot-based spatiotemporal attention and memory buffers, resulting in significant improvements in object segmentation, tracking, and downstream tasks like VQA across various datasets.
Object-centric learning aims to break down complex visual scenes into more manageable object representations, enhancing the understanding and reasoning abilities of machine learning systems toward the physical world. Recently, slot-based video models have demonstrated remarkable proficiency in segmenting and tracking objects, but they overlook the importance of the effective reasoning module. In the real world, reasoning and predictive abilities play a crucial role in human perception and object tracking; in particular, these abilities are closely related to human intuitive physics. Inspired by this, we designed a novel reasoning module called the Slot-based Time-Space Transformer with Memory buffer (STATM) to enhance the model's perception ability in complex scenes. The memory buffer primarily serves as storage for slot information from upstream modules, the Slot-based Time-Space Transformer makes predictions through slot-based spatiotemporal attention computations and fusion. Our experimental results on various datasets indicate that the STATM module can significantly enhance the capabilities of multiple state-of-the-art object-centric learning models for video. Moreover, as a predictive model, the STATM module also performs well in downstream prediction and Visual Question Answering (VQA) tasks. We will release our codes and data at https://github.com/intell-sci-comput/STATM.