Towards Automated Identification of Violation Symptoms of Architecture Erosion

Architecture erosion harms software maintenance when implementation deviates from intended architecture. Early detection of violation symptoms is essential, and code review comments provide a promising textual source for automated identification. This paper investigates automatic detection of architecture violation symptoms from code reviews using traditional machine learning (ML), deep learning (DL), and large language models (LLMs). We built and evaluated 15 ML and 4 DL classifiers with word embeddings (word2vec, fastText, GloVe) on code reviews from four open-source projects (OpenStack Nova, Neutron, Qt Base, Creator). We also constructed LLM-based classifiers using GPT-4o, Qwen-2.5, and DeepSeek-R1. Results show that SVM with word2vec performs best among ML/DL methods (F1-score: 0.808), with 200-dimensional embeddings generally superior. Majority-vote ensembles further improve the performance. On an imbalanced test set, SVM remains strongest in Macro-F1. Statistical tests confirm significant performance differences. The best LLM-based classifiers outperformed traditional ML/DL approaches, with GPT-4o achieving the highest F1-score (0.851). Practitioner surveys and interviews indicate practical value for early warning of architectural issues. A controlled experiment shows that providing detected violation symptoms improves developers' detection rate from 25.9% to 64.7%. The study contributes an automated detection approach, comprehensive ML/DL vs LLM comparison, and practitioner-validated evidence for improving architectural conformance and sustainability.