Di Chen

Amritanshu Agrawal, Wei Fu, Di Chen et al.

Machine learning techniques applied to software engineering tasks can be improved by hyperparameter optimization, i.e., automatic tools that find good settings for a learner's control parameters. We show that such hyperparameter optimization can be unnecessarily slow, particularly when the optimizers waste time exploring "redundant tunings"', i.e., pairs of tunings which lead to indistinguishable results. By ignoring redundant tunings, DODGE, a tuning tool, runs orders of magnitude faster, while also generating learners with more accurate predictions than seen in prior state-of-the-art approaches.

Di Chen, Wei Fu, Rahul Krishna et al.

Actionable analytics are those that humans can understand, and operationalize. What kind of data mining models generate such actionable analytics? According to psychological scientists, humans understand models that most match their own internal models, which they characterize as lists of "heuristic" (i.e., lists of very succinct rules). One such heuristic rule generator is the Fast-and-Frugal Trees (FFT) preferred by psychological scientists. Despite their successful use in many applied domains, FFTs have not been applied in software analytics. Accordingly, this paper assesses FFTs for software analytics. We find that FFTs are remarkably effective. Their models are very succinct (5 lines or less describing a binary decision tree). These succinct models outperform state-of-the-art defect prediction algorithms defined by Ghortra et al. at ICSE'15. Also, when we restrict training data to operational attributes (i.e., those attributes that are frequently changed by developers), FFTs perform much better than standard learners. Our conclusions are two-fold. Firstly, there is much that software analytics community could learn from psychological science. Secondly, proponents of complex methods should always baseline those methods against simpler alternatives. For example, FFTs could be used as a standard baseline learner against which other software analytics tools are compared.

Wei Fu, Tim Menzies, Di Chen et al.

Despite extensive research, many methods in software quality prediction still exhibit some degree of uncertainty in their results. Rather than treating this as a problem, this paper asks if this uncertainty is a resource that can simplify software quality prediction. For example, Deb's principle of $ε$-dominance states that if there exists some $ε$ value below which it is useless or impossible to distinguish results, then it is superfluous to explore anything less than $ε$. We say that for "large $ε$ problems", the results space of learning effectively contains just a few regions. If many learners are then applied to such large $ε$ problems, they would exhibit a "many roads lead to Rome" property; i.e., many different software quality prediction methods would generate a small set of very similar results. This paper explores DART, an algorithm especially selected to succeed for large $ε$ software quality prediction problems. DART is remarkable simple yet, on experimentation, it dramatically out-performs three sets of state-of-the-art defect prediction methods. The success of DART for defect prediction begs the questions: how many other domains in software quality predictors can also be radically simplified? This will be a fruitful direction for future work.

Di Chen, Kathryn T. Stolee, Tim Menzies

Due to the difficulties in replicating and scaling up qualitative studies, such studies are rarely verified. Accordingly, in this paper, we leverage the advantages of crowdsourcing (low costs, fast speed, scalable workforce) to replicate and scale-up one state-of-the-art qualitative study. That qualitative study explored 20 GitHub pull requests to learn factors that influence the fate of pull requests with respect to approval and merging. As a secondary study, using crowdsourcing at a cost of $200, we studied 250 pull requests from 142 GitHub projects. The prior qualitative findings are mapped into questions for crowds workers. Their answers were converted into binary features to build a predictor which predicts whether code would be merged with median F1 scores of 68%. For the same large group of pull requests, the median F1 scores could achieve 90% by a predictor built with additional features defined by prior quantitative results. Based on this case study, we conclude that there is much benefit in combining different kinds of research methods. While qualitative insights are very useful for finding novel insights, they can be hard to scale or replicate. That said, they can guide and define the goals of scalable secondary studies that use (e.g.) crowdsourcing+data mining. On the other hand, while data mining methods are reproducible and scalable to large data sets, their results may be spectacularly wrong since they lack contextual information. That said, they can be used to test the stability and external validity, of the insights gained from a qualitative analysis.