OSS-UAgent: An Agent-based Usability Evaluation Framework for Open Source Software
This addresses usability evaluation challenges for open source software developers and maintainers, representing a novel application of agent-based methods in this domain.
The paper tackles the high cost and limited scalability of traditional usability evaluation methods for open source software by introducing OSS-UAgent, an automated agent-based framework that uses LLMs to simulate developers and evaluate code across multiple dimensions, demonstrating its effectiveness on graph analytics platforms.
Usability evaluation is critical to the impact and adoption of open source software (OSS), yet traditional methods relying on human evaluators suffer from high costs and limited scalability. To address these limitations, we introduce OSS-UAgent, an automated, configurable, and interactive agent-based usability evaluation framework specifically designed for open source software. Our framework employs intelligent agents powered by large language models (LLMs) to simulate developers performing programming tasks across various experience levels (from Junior to Expert). By dynamically constructing platform-specific knowledge bases, OSS-UAgent ensures accurate and context-aware code generation. The generated code is automatically evaluated across multiple dimensions, including compliance, correctness, and readability, providing a comprehensive measure of the software's usability. Additionally, our demonstration showcases OSS-UAgent's practical application in evaluating graph analytics platforms, highlighting its effectiveness in automating usability evaluation.