Automating construction safety inspections using a multi-modal vision-language RAG framework
This addresses inefficiencies in construction safety inspections for industry practitioners, though it appears incremental as it builds on existing LVLM and RAG methods.
The study tackled the problem of inefficient construction safety inspections by introducing SiteShield, a multi-modal vision-language RAG framework that integrates visual and audio inputs, achieving an F1 score of 0.82 and recall of 0.96 on real-world data.
Conventional construction safety inspection methods are often inefficient as they require navigating through large volume of information. Recent advances in large vision-language models (LVLMs) provide opportunities to automate safety inspections through enhanced visual and linguistic understanding. However, existing applications face limitations including irrelevant or unspecific responses, restricted modal inputs and hallucinations. Utilisation of Large Language Models (LLMs) for this purpose is constrained by availability of training data and frequently lack real-time adaptability. This study introduces SiteShield, a multi-modal LVLM-based Retrieval-Augmented Generation (RAG) framework for automating construction safety inspection reports by integrating visual and audio inputs. Using real-world data, SiteShield outperformed unimodal LLMs without RAG with an F1 score of 0.82, hamming loss of 0.04, precision of 0.76, and recall of 0.96. The findings indicate that SiteShield offers a novel pathway to enhance information retrieval and efficiency in generating safety reports.