Supporting architecture evaluation for ATAM scenarios with LLMs
This work addresses the inefficiency in software architecture evaluation for stakeholders by offering an incremental approach to automate scenario analysis.
The paper tackles the problem of manual, time-consuming architecture evaluation in software design by proposing the use of Large Language Models (LLMs) to partially automate the assessment and selection of quality-attribute scenarios, with an initial study showing that an LLM (MS Copilot) produced better and more accurate results in most cases compared to students' evaluations.
Architecture evaluation methods have long been used to evaluate software designs. Several evaluation methods have been proposed and used to analyze tradeoffs between different quality attributes. Having competing qualities leads to conflicts for selecting which quality-attribute scenarios are the most suitable ones that an architecture should tackle and for prioritizing the scenarios required by the stakeholders. In this context, architecture evaluation is carried out manually, often involving long brainstorming sessions to decide which are the most adequate quality scenarios. To reduce this effort and make the assessment and selection of scenarios more efficient, we suggest the usage of LLMs to partially automate evaluation activities. As a first step to validate this hypothesis, this work studies MS Copilot as an LLM tool to analyze quality scenarios suggested by students in a software architecture course and compares the students' results with the assessment provided by the LLM. Our initial study reveals that the LLM produces in most cases better and more accurate results regarding the risks, sensitivity points and tradeoff analysis of the quality scenarios. Overall, the use of generative AI has the potential to partially automate and support the architecture evaluation tasks, improving the human decision-making process.