Tracking Without Re-recognition in Humans and Machines
This work addresses a fundamental limitation in computer vision for object tracking, particularly in scenarios with visual nuisances, by bridging insights from human vision to enhance machine performance.
The paper tackled the problem of tracking objects when appearance cues are unreliable, by introducing a synthetic challenge (PathTracker) where both humans and machines track a target among identical distractors. The result was a recurrent circuit model based on biological motion mechanisms that achieved human-level performance on PathTracker and improved state-of-the-art on TrackingNet, with a new SOTA performance.
Imagine trying to track one particular fruitfly in a swarm of hundreds. Higher biological visual systems have evolved to track moving objects by relying on both appearance and motion features. We investigate if state-of-the-art deep neural networks for visual tracking are capable of the same. For this, we introduce PathTracker, a synthetic visual challenge that asks human observers and machines to track a target object in the midst of identical-looking "distractor" objects. While humans effortlessly learn PathTracker and generalize to systematic variations in task design, state-of-the-art deep networks struggle. To address this limitation, we identify and model circuit mechanisms in biological brains that are implicated in tracking objects based on motion cues. When instantiated as a recurrent network, our circuit model learns to solve PathTracker with a robust visual strategy that rivals human performance and explains a significant proportion of their decision-making on the challenge. We also show that the success of this circuit model extends to object tracking in natural videos. Adding it to a transformer-based architecture for object tracking builds tolerance to visual nuisances that affect object appearance, resulting in a new state-of-the-art performance on the large-scale TrackingNet object tracking challenge. Our work highlights the importance of building artificial vision models that can help us better understand human vision and improve computer vision.