Assessing the Auditability of AI-integrating Systems: A Framework and Learning Analytics Case Study
This addresses the need for trustworthy audits in AI-based Learning Analytics, which is crucial for educators and regulators, though it is incremental as it builds on existing audit concepts.
The paper tackles the problem of ensuring AI-integrating systems are auditable by proposing a framework with three components—verifiable claims, evidence types, and accessibility—and applies it to Learning Analytics systems like Moodle, finding limitations such as incomplete documentation and insufficient monitoring.
Audits contribute to the trustworthiness of Learning Analytics (LA) systems that integrate Artificial Intelligence (AI) and may be legally required in the future. We argue that the efficacy of an audit depends on the auditability of the audited system. Therefore, systems need to be designed with auditability in mind. We present a framework for assessing the auditability of AI-integrating systems that consists of three parts: (1) Verifiable claims about the validity, utility and ethics of the system, (2) Evidence on subjects (data, models or the system) in different types (documentation, raw sources and logs) to back or refute claims, (3) Evidence must be accessible to auditors via technical means (APIs, monitoring tools, explainable AI, etc.). We apply the framework to assess the auditability of Moodle's dropout prediction system and a prototype AI-based LA. We find that Moodle's auditability is limited by incomplete documentation, insufficient monitoring capabilities and a lack of available test data. The framework supports assessing the auditability of AI-based LA systems in use and improves the design of auditable systems and thus of audits.