Multi-Object Tracking and Segmentation via Neural Message Passing
This work addresses the problem of improving accuracy and efficiency in MOTS for computer vision applications, representing an incremental advancement by applying neural message passing to an existing network flow formulation.
The paper tackles the challenge of learning models for multiple object tracking and segmentation (MOTS) in structured graph domains by introducing a fully differentiable framework based on Message Passing Networks (MPNs), achieving state-of-the-art results in tracking and segmentation on several datasets.
Graphs offer a natural way to formulate Multiple Object Tracking (MOT) and Multiple Object Tracking and Segmentation (MOTS) within the tracking-by-detection paradigm. However, they also introduce a major challenge for learning methods, as defining a model that can operate on such structured domain is not trivial. In this work, we exploit the classical network flow formulation of MOT to define a fully differentiable framework based on Message Passing Networks (MPNs). By operating directly on the graph domain, our method can reason globally over an entire set of detections and exploit contextual features. It then jointly predicts both final solutions for the data association problem and segmentation masks for all objects in the scene while exploiting synergies between the two tasks. We achieve state-of-the-art results for both tracking and segmentation in several publicly available datasets. Our code is available at github.com/ocetintas/MPNTrackSeg.