CAPTURE: A Benchmark and Evaluation for LVLMs in CAPTCHA Resolving
This addresses the lack of comprehensive benchmarks for LVLMs in CAPTCHA resolution, though it appears incremental as it extends existing benchmarking approaches to a new domain.
The authors introduced CAPTURE, the first dedicated benchmark for evaluating Large Visual Language Models (LVLMs) on CAPTCHA solving, covering 4 main types and 25 sub-types from 31 vendors, and found that current LVLMs perform poorly on this task.
Benefiting from strong and efficient multi-modal alignment strategies, Large Visual Language Models (LVLMs) are able to simulate human visual and reasoning capabilities, such as solving CAPTCHAs. However, existing benchmarks based on visual CAPTCHAs still face limitations. Previous studies, when designing benchmarks and datasets, customized them according to their research objectives. Consequently, these benchmarks cannot comprehensively cover all CAPTCHA types. Notably, there is a dearth of dedicated benchmarks for LVLMs. To address this problem, we introduce a novel CAPTCHA benchmark for the first time, named CAPTURE CAPTCHA for Testing Under Real-world Experiments, specifically for LVLMs. Our benchmark encompasses 4 main CAPTCHA types and 25 sub-types from 31 vendors. The diversity enables a multi-dimensional and thorough evaluation of LVLM performance. CAPTURE features extensive class variety, large-scale data, and unique LVLM-tailored labels, filling the gaps in previous research in terms of data comprehensiveness and labeling pertinence. When evaluated by this benchmark, current LVLMs demonstrate poor performance in solving CAPTCHAs.