IDSelect: A RL-Based Cost-Aware Selection Agent for Video-based Multi-Modal Person Recognition
This work addresses computational waste in video-based person recognition systems, offering a cost-effective solution for real-world applications, though it is incremental as it builds on existing multimodal recognition methods.
The paper tackles the inefficiency of fixed multimodal ensembles in video-based person recognition by proposing IDSelect, a reinforcement learning-based selector that chooses one pre-trained model per modality per sequence to optimize accuracy and computational cost. On the CCVID dataset, IDSelect achieved 95.9% Rank-1 accuracy with 92.4% less computation than baselines and a 1.8% accuracy improvement, and on MEVID, it reduced computation by 41.3% while maintaining competitive performance.
Video-based person recognition achieves robust identification by integrating face, body, and gait. However, current systems waste computational resources by processing all modalities with fixed heavyweight ensembles regardless of input complexity. To address these limitations, we propose IDSelect, a reinforcement learning-based cost-aware selector that chooses one pre-trained model per modality per-sequence to optimize the accuracy-efficiency trade-off. Our key insight is that an input-conditioned selector can discover complementary model choices that surpass fixed ensembles while using substantially fewer resources. IDSelect trains a lightweight agent end-to-end using actor-critic reinforcement learning with budget-aware optimization. The reward balances recognition accuracy with computational cost, while entropy regularization prevents premature convergence. At inference, the policy selects the most probable model per modality and fuses modality-specific similarities for the final score. Extensive experiments on challenging video-based datasets demonstrate IDSelect's superior efficiency: on CCVID, it achieves 95.9% Rank-1 accuracy with 92.4% less computation than strong baselines while improving accuracy by 1.8%; on MEVID, it reduces computation by 41.3% while maintaining competitive performance.