Beyond Walking: A Large-Scale Image-Text Benchmark for Text-based Person Anomaly Search
This addresses the need for identifying abnormal pedestrian behaviors in real-world surveillance scenarios, representing an incremental advancement in person search.
The paper tackles the problem of text-based person search being biased toward common actions by introducing a new task for identifying both routine and anomalous pedestrian behaviors via text descriptions. They created a large-scale Pedestrian Anomaly Behavior benchmark with over 1 million synthetic training pairs and nearly 2,000 real-world test pairs, and their pose-aware method achieved 84.93% recall@1 accuracy.
Text-based person search aims to retrieve specific individuals across camera networks using natural language descriptions. However, current benchmarks often exhibit biases towards common actions like walking or standing, neglecting the critical need for identifying abnormal behaviors in real-world scenarios. To meet such demands, we propose a new task, text-based person anomaly search, locating pedestrians engaged in both routine or anomalous activities via text. To enable the training and evaluation of this new task, we construct a large-scale image-text Pedestrian Anomaly Behavior (PAB) benchmark, featuring a broad spectrum of actions, e.g., running, performing, playing soccer, and the corresponding anomalies, e.g., lying, being hit, and falling of the same identity. The training set of PAB comprises 1,013,605 synthesized image-text pairs of both normalities and anomalies, while the test set includes 1,978 real-world image-text pairs. To validate the potential of PAB, we introduce a cross-modal pose-aware framework, which integrates human pose patterns with identity-based hard negative pair sampling. Extensive experiments on the proposed benchmark show that synthetic training data facilitates the fine-grained behavior retrieval, and the proposed pose-aware method arrives at 84.93% recall@1 accuracy, surpassing other competitive methods. The dataset, model, and code are available at https://github.com/Shuyu-XJTU/CMP.