Ye Yang

Lin Shi, Xiao Chen, Ye Yang et al.

Modern communication platforms such as Gitter and Slack play an increasingly critical role in supporting software teamwork, especially in open source development.Conversations on such platforms often contain intensive, valuable information that may be used for better understanding OSS developer communication and collaboration. However, little work has been done in this regard. To bridge the gap, this paper reports a first comprehensive empirical study on developers' live chat, investigating when they interact, what community structures look like, which topics are discussed, and how they interact. We manually analyze 749 dialogs in the first phase, followed by an automated analysis of over 173K dialogs in the second phase. We find that developers tend to converse more often on weekdays, especially on Wednesdays and Thursdays (UTC), that there are three common community structures observed, that developers tend to discuss topics such as API usages and errors, and that six dialog interaction patterns are identified in the live chat communities. Based on the findings, we provide recommendations for individual developers and OSS communities, highlight desired features for platform vendors, and shed light on future research directions. We believe that the findings and insights will enable a better understanding of developers' live chat, pave the way for other researchers, as well as a better utilization and mining of knowledge embedded in the massive chat history.

Razieh Saremi, Ye Yang, Gregg Vesonder et al.

A typical crowdsourcing software development(CSD) marketplace consists of a list of software tasks as service demands and a pool of freelancer developers as service suppliers. Highly dynamic and competitive CSD market places may result in task failure due to unforeseen risks, such as increased competition over shared worker supply, or uncertainty associated with workers' experience and skills, and so on. To improve CSD effectiveness, it is essential to better understand and plan with respect to dynamic worker characteristics and risks associated with CSD processes. In this paper, we present a hybrid simulation model, CrowdSim, to forecast crowdsourcing task failure risk in competitive CSD platforms. CrowdSim is composed of three layered components: the macro-level reflects the overall crowdsourcing platform based on system dynamics,the meso-level represents the task life cycle based on discrete event simulation, and the micro-level models the crowd workers' decision-making processes based on agent-based simulation. CrowdSim is evaluated through three CSD decision scenarios to demonstrate its effectiveness, using a real-world historical dataset and the results demonstrate CrowdSim's potential in empowering crowdsourcing managers to explore crowdsourcing outcomes with respect to different task scheduling options.

Denisse Martinez Mejorado, Razieh Saremi, Ye Yang et al.

Context: The success of software crowdsourcing depends on steady tasks supply and active worker pool. Existing analysis reveals an average task failure ratio of 15.7% in software crowdsourcing market. Goal: The objective of this study is to empirically investigate patterns and effect of task diversity in software crowdsourcing platform in order to improve the success and efficiency of software crowdsourcing. Method: We propose a conceptual task diversity model, and develop an approach to measuring and analyzing task diversity.More specifically, this includes grouping similar tasks, ranking them based on their competition level and identifying the dominant attributes that distinguish among these levels, and then studying the impact of task diversity on task success and worker performance in crowdsourcing platform. The empirical study is conducted on more than one year's real-world data from TopCoder, the leading software crowdsourcing platform. Results: We identified that monetary prize and task complexity are the dominant attributes that differentiate among different competition levels. Based on these dominant attributes, we found three task diversity patterns (configurations) from workers behavior perspective: responsive to prize, responsive to prize and complexity and over responsive to prize. This study supports that1) responsive to prize configuration provides highest level of task density and workers' reliability in a platform; 2) responsive to prize and complexity configuration leads to attracting high level of trustworthy workers; 3) over responsive to prize configuration results in highest task stability and the lowest failure ratio in the platform for not high similar tasks.

Junjie Wang, Ye Yang, Zhe Yu et al.

Trade-offs such as "how much testing is enough" are critical yet challenging project decisions in software engineering. Most existing approaches adopt risk-driven or value-based analysis to prioritize test cases and minimize test runs. However, none of these is applicable to the emerging crowd testing paradigm where task requesters typically have no control over online crowdworkers's dynamic behavior and uncertain performance. In current practice, deciding when to close a crowdtesting task is largely done by guesswork due to lack of decision support. This paper intends to fill this gap by introducing automated decision support for monitoring and determining appropriate time to close the crowdtesting tasks. First, this paper investigates the necessity and feasibility of close prediction of crowdtesting tasks based on industrial dataset. Then,it designs 8 methods for close prediction, based on various models including the bug trend, bug arrival model, capture-recapture model.Finally, the evaluation is conducted on 218 crowdtesting tasks from one of the largest crowdtesting platforms in China, and the results show that a median of 91% bugs can be detected with 49% saved cost.

Junjie Wang, Ye Yang, Rahul Krishna et al.

Crowdtesting has grown to be an effective alter-native to traditional testing, especially in mobile apps. However,crowdtesting is hard to manage in nature. Given the complexity of mobile applications and unpredictability of distributed, parallel crowdtesting process, it is difficult to estimate (a) the remaining number of bugs as yet undetected or (b) the required cost to find those bugs. Experience-based decisions may result in ineffective crowdtesting process. This paper aims at exploring automated decision support to effectively manage crowdtesting process. The proposed ISENSE applies incremental sampling technique to process crowdtesting reports arriving in chronological order, organizes them into fixed-size groups as dynamic inputs, and predicts two test completion indicators in an incrementally manner. The two indicators are: 1)total number of bugs predicted with Capture-ReCapture (CRC)model, and 2) required test cost for achieving certain test objectives predicted with AutoRegressive Integrated Moving Average(ARIMA) model. We assess ISENSE using 46,434 reports of 218 crowdtesting tasks from one of the largest crowdtesting platforms in China. Its effectiveness is demonstrated through two applications for automating crowdtesting management, i.e. automation oftask closing decision, and semi-automation of task closing trade-off analysis. The results show that decision automation using ISENSE will provide managers with greater opportunities to achieve cost-effectiveness gains of crowdtesting. Specifically, a median of 100% bugs can be detected with 30% saved cost basedon the automated close prediction

Muhammad Rezaul Karim, David Messinger, Ye Yang et al.

Crowdsourced software development (CSD) offers a series of specified tasks to a large crowd of trustworthy software workers. Topcoder is a leading platform to manage the whole process of CSD. While increasingly accepted as a realistic option for software development, preliminary analysis on Topcoder's software crowd worker behaviors reveals an alarming task-quitting rate of 82.9%. In addition, a substantial number of tasks do not receive any successful submission. In this paper, we report about a methodology to improve the efficiency of CSD. We apply massive data analytics and machine leaning to (i) perform comparative analysis on alternative technique analysis to predict likelihood of winners and quitters for each task, (ii) significantly reduce the amount of non-succeeding development effort in registered but inappropriate tasks, (iii) identify and rank the most qualified registered workers for each task, and (iv) provide reliable prediction of tasks risky to get any successful submission. Our results and analysis show that Random Forest (RF) based predictive technique performs best among the alternative techniques studied. Applying RF, the tasks recommended to workers can reduce the amount of non-succeeding development effort to a great extent. On average, over a period of 30 days, the savings are 3.5 and 4.6 person-days per registered tasks for experienced resp. unexperienced workers. For the task-related recommendations of workers, we can accurately recommend at least 1 actual winner in the top ranked workers, particularly 94.07% of the time among the top-2 recommended workers for each task. Finally, we can predict, with more than 80% F-measure, the tasks likely not getting any submission, thus triggering timely corrective actions from CSD platforms or task requesters.

Yueming Sun, Ye Yang, He Zhang et al.

[Background]: Systematic Literature Review (SLR) has become an important software engineering research method but costs tremendous efforts. [Aim]: This paper proposes an approach to leverage on empirically evolved ontology to support automating key SLR activities. [Method]: First, we propose an ontology, SLRONT, built on SLR experiences and best practices as a groundwork to capture common terminologies and their relationships during SLR processes; second, we present an extended version of SLRONT, the COSONT and instantiate it with the knowledge and concepts extracted from structured abstracts. Case studies illustrate the details of applying it for supporting SLR steps. [Results]: Results show that through using COSONT, we acquire the same conclusion compared with sheer manual works, but the efforts involved is significantly reduced. [Conclusions]: The approach of using ontology could effectively and efficiently support the conducting of systematic literature review.

Tim Menzies, Ye Yang, George Mathew et al.

Context:More than half the literature on software effort estimation (SEE) focuses on comparisons of new estimation methods. Surprisingly, there are no studies comparing state of the art latest methods with decades-old approaches. Objective:To check if new SEE methods generated better estimates than older methods. Method: Firstly, collect effort estimation methods ranging from "classical" COCOMO (parametric estimation over a pre-determined set of attributes) to "modern" (reasoning via analogy using spectral-based clustering plus instance and feature selection, and a recent "baseline method" proposed in ACM Transactions on Software Engineering).Secondly, catalog the list of objections that lead to the development of post-COCOMO estimation methods.Thirdly, characterize each of those objections as a comparison between newer and older estimation methods.Fourthly, using four COCOMO-style data sets (from 1991, 2000, 2005, 2010) and run those comparisons experiments.Fifthly, compare the performance of the different estimators using a Scott-Knott procedure using (i) the A12 effect size to rule out "small" differences and (ii) a 99% confident bootstrap procedure to check for statistically different groupings of treatments). Results: The major negative results of this paper are that for the COCOMO data sets, nothing we studied did any better than Boehm's original procedure. Conclusions: When COCOMO-style attributes are available, we strongly recommend (i) using that data and (ii) use COCOMO to generate predictions. We say this since the experiments of this paper show that, at least for effort estimation,how data is collected is more important than what learner is applied to that data.