LLM attribution analysis across different fine-tuning strategies and model scales for automated code compliance
For researchers and practitioners in the Architecture, Engineering, and Construction industry, this work provides insights into the explainability of LLMs for critical regulation-based tasks.
This paper uses perturbation-based attribution analysis to compare the interpretive behaviors of LLMs fine-tuned for automated code compliance, finding that full fine-tuning yields more focused attribution patterns than parameter-efficient methods, and that model scale increases lead to specific interpretive strategies but performance gains plateau beyond 7B parameters.
Existing research on large language models (LLMs) for automated code compliance has primarily focused on performance, treating the models as black boxes and overlooking how training decisions affect their interpretive behavior. This paper addresses this gap by employing a perturbation-based attribution analysis to compare the interpretive behaviors of LLMs across different fine-tuning strategies such as full fine-tuning (FFT), low-rank adaptation (LoRA) and quantized LoRA fine-tuning, as well as the impact of model scales which include varying LLM parameter sizes. Our results show that FFT produces attribution patterns that are statistically different and more focused than those from parameter-efficient fine-tuning methods. Furthermore, we found that as model scale increases, LLMs develop specific interpretive strategies such as prioritizing numerical constraints and rule identifiers in the building text, albeit with performance gains in semantic similarity of the generated and reference computer-processable rules plateauing for models larger than 7B. This paper provides crucial insights into the explainability of these models, taking a step toward building more transparent LLMs for critical, regulation-based tasks in the Architecture, Engineering, and Construction industry.