5.1CYOct 25, 2024
Can We Trust AI Agents? A Case Study of an LLM-Based Multi-Agent System for Ethical AIJosé Antonio Siqueira de Cerqueira, Mamia Agbese, Rebekah Rousi et al.
AI-based systems, including Large Language Models (LLM), impact millions by supporting diverse tasks but face issues like misinformation, bias, and misuse. AI ethics is crucial as new technologies and concerns emerge, but objective, practical guidance remains debated. This study examines the use of LLMs for AI ethics in practice, assessing how LLM trustworthiness-enhancing techniques affect software development in this context. Using the Design Science Research (DSR) method, we identify techniques for LLM trustworthiness: multi-agents, distinct roles, structured communication, and multiple rounds of debate. We design a multi-agent prototype LLM-MAS, where agents engage in structured discussions on real-world AI ethics issues from the AI Incident Database. We evaluate the prototype across three case scenarios using thematic analysis, hierarchical clustering, comparative (baseline) studies, and running source code. The system generates approximately 2,000 lines of code per case, compared to only 80 lines in baseline trials. Discussions reveal terms like bias detection, transparency, accountability, user consent, GDPR compliance, fairness evaluation, and EU AI Act compliance, showing this prototype ability to generate extensive source code and documentation addressing often overlooked AI ethics issues. However, practical challenges in source code integration and dependency management may limit its use by practitioners.
3.7HCJun 18, 2021
Do people's user types change over time? An exploratory studyAna Cláudia Guimarães Santos, Wilk Oliveira, Juho Hamari et al.
In recent years, different studies have proposed and validated user models (e.g., Bartle, BrainHex, and Hexad) to represent the different user profiles in games and gamified settings. However, the results of applying these user models in practice (e.g., to personalize gamified systems) are still contradictory. One of the hypotheses for these results is that the user types can change over time (i.e., user types are dynamic). To start to understand whether user types can change over time, we conducted an exploratory study analyzing data from 74 participants to identify if their user type (Achiever, Philanthropist, Socialiser, Free Spirit, Player, and Disruptor) had changed over time (six months). The results indicate that there is a change in the dominant user type of the participants, as well as the average scores in the Hexad sub-scales. These results imply that all the scores should be considered when defining the Hexad's user type and that the user types are dynamic. Our results contribute with practical implications, indicating that the personalization currently made (generally static) may be insufficient to improve the users' experience, requiring user types to be analyzed continuously and personalization to be done dynamically.
3.7HCJun 18, 2021
Does gamification affect flow experience? A systematic literature reviewWilk Oliveira, Olena Pastushenko, Luiz Rodrigues et al.
In recent years, studies in different areas have used gamification to improve users' flow experience. However, due to the high variety of the conducted studies and the lack of secondary studies (e.g., systematic literature reviews) in this field, it is difficult to get the state-of-the-art of this research domain. To address this problem, we conducted a systematic literature review to identify i) which gamification design methods have been used in the studies about gamification and Flow Theory, ii) which gamification elements have been used in these studies, iii) which methods have been used to evaluate the users' flow experience in gamified settings, and iv) how gamification affects users' flow experience. The main results show that there is growing interest to this field, as the number of publications is increasing. The most significant interest is in the area of gamification in education. However, there is no unanimity regarding the preferred method of the study or the effects of gamification on users' experience. Our results highlight the importance of conducting new experimental studies investigating how gamification affects the users' flow experience in different gamified settings, applications and domains.