AI Failures in the Eyes of the Downstream Developer: A First Look at Concerns, Practices, and Challenges
For researchers and tool builders, it highlights a gap between technical AI failure taxonomies and developers' practical awareness, suggesting a need for better support in real-world AI integration.
This study investigates how downstream developers perceive and handle AI failures when reusing pre-trained models, revealing that they prioritize functional correctness over ethical concerns like bias or data leakage, and face challenges due to lack of documentation and tooling.
With the advancement of AI models, more software systems are adopting AI as a component to facilitate automation. Pre-trained models (PTMs) have become a cornerstone of AI-based software, allowing for rapid integration and development with lower training cost. However, their adoption also introduces failure modes such as data leakage and biased outputs, that may require careful handling by downstream developers. While previous research has proposed taxonomies of these technical concerns and various mitigation strategies, how downstream developers address these issues during the development of general AI-based software when reusing PTMs remains unexplored. Understanding downstream developers' perspectives is essential because they directly influence how these potential failures concerns translate into practice, such as determining whether immediate risks like data leakage or model bias are recognised, mitigated, or inadvertently overlooked in real-world deployments. This study investigates downstream developers' concerns, practices and perceived challenges regarding practical AI failures during the development of AI-based software. To achieve this, we conducted a mixed-method study, including interviews with 16 participants, a survey of 86 practitioners,