FeedbackLogs: Recording and Incorporating Stakeholder Feedback into Machine Learning Pipelines
This addresses the need for better documentation and accountability in ML pipelines for stakeholders, though it is incremental as it builds on existing documentation practices.
The paper tackles the problem of recording and incorporating stakeholder feedback in machine learning pipelines by proposing FeedbackLogs, which track feedback collection, content, and usage, and demonstrates their application in algorithmic auditing and pipeline updates.
Even though machine learning (ML) pipelines affect an increasing array of stakeholders, there is little work on how input from stakeholders is recorded and incorporated. We propose FeedbackLogs, addenda to existing documentation of ML pipelines, to track the input of multiple stakeholders. Each log records important details about the feedback collection process, the feedback itself, and how the feedback is used to update the ML pipeline. In this paper, we introduce and formalise a process for collecting a FeedbackLog. We also provide concrete use cases where FeedbackLogs can be employed as evidence for algorithmic auditing and as a tool to record updates based on stakeholder feedback.