Bad Smells in Software Analytics Papers
This addresses concerns about reliability in software engineering analytics for researchers and practitioners, but it is incremental as it builds on existing concepts from agile software development.
The paper tackles the problem of unreliable software analytics studies by proposing 12 'bad smells' as indicators of deeper issues, aiming to guide producers and consumers in improving study validity.
CONTEXT: There has been a rapid growth in the use of data analytics to underpin evidence-based software engineering. However the combination of complex techniques, diverse reporting standards and poorly understood underlying phenomena are causing some concern as to the reliability of studies. OBJECTIVE: Our goal is to provide guidance for producers and consumers of software analytics studies (computational experiments and correlation studies). METHOD: We propose using "bad smells", i.e., surface indications of deeper problems and popular in the agile software community and consider how they may be manifest in software analytics studies. RESULTS: We list 12 "bad smells" in software analytics papers (and show their impact by examples). CONCLUSIONS: We believe the metaphor of bad smell is a useful device. Therefore we encourage more debate on what contributes to the validty of software analytics studies (so we expect our list will mature over time).