LVLMs as inspectors: an agentic framework for category-level structural defect annotation
This provides a scalable, cost-effective solution for infrastructure safety by automating defect labeling, though it is incremental as it builds on existing LVLM methods.
The paper tackles automated structural defect annotation by introducing the ADPT framework, which uses LVLMs and iterative refinement to achieve up to 98% accuracy in defect detection and 85%-98% annotation accuracy across categories without manual supervision.
Automated structural defect annotation is essential for ensuring infrastructure safety while minimizing the high costs and inefficiencies of manual labeling. A novel agentic annotation framework, Agent-based Defect Pattern Tagger (ADPT), is introduced that integrates Large Vision-Language Models (LVLMs) with a semantic pattern matching module and an iterative self-questioning refinement mechanism. By leveraging optimized domain-specific prompting and a recursive verification process, ADPT transforms raw visual data into high-quality, semantically labeled defect datasets without any manual supervision. Experimental results demonstrate that ADPT achieves up to 98% accuracy in distinguishing defective from non-defective images, and 85%-98% annotation accuracy across four defect categories under class-balanced settings, with 80%-92% accuracy on class-imbalanced datasets. The framework offers a scalable and cost-effective solution for high-fidelity dataset construction, providing strong support for downstream tasks such as transfer learning and domain adaptation in structural damage assessment.