Towards evaluating and eliciting high-quality documentation for intelligent systems
This work addresses the problem of defining and evaluating high-quality documentation for intelligent systems, which is crucial for improving trust and transparency for users and developers.
This paper proposes and evaluates a set of quality dimensions for documentation of intelligent systems. It then uses these dimensions to evaluate three different approaches for eliciting such documentation, demonstrating their effectiveness in identifying shortcomings.
A vital component of trust and transparency in intelligent systems built on machine learning and artificial intelligence is the development of clear, understandable documentation. However, such systems are notorious for their complexity and opaqueness making quality documentation a non-trivial task. Furthermore, little is known about what makes such documentation "good." In this paper, we propose and evaluate a set of quality dimensions to identify in what ways this type of documentation falls short. Then, using those dimensions, we evaluate three different approaches for eliciting intelligent system documentation. We show how the dimensions identify shortcomings in such documentation and posit how such dimensions can be use to further enable users to provide documentation that is suitable to a given persona or use case.