Razieh Saremi

Razieh Saremi

Open software development provides software organizations access to infinite online resource supply. The resource supply is a pool of unknown workers who work from different location and time zone and are interested in performing various type of tasks. Improper task execution in such dynamic and competitive environment leads to zero task registration, zero task submissions or low qualified submissions due to unforeseen reasons such as uncertainty in workers' behavior and performance. Therefore, to ensure effectiveness of open software development, there is a need for improved understanding and visibility into characteristics associated with attracting reliable workers in making qualified submissions and reducing task failure.

Hamid Shamszare, Razieh Saremi, Sanam Jena

The success of software crowdsourcing depends on active and trustworthy pool of worker supply. The uncertainty of crowd workers' behaviors makes it challenging to predict workers' success and plan accordingly. In a competitive crowdsourcing marketplace, competition for success over shared tasks adds another layer of uncertainty in crowd workers' decision-making process. Preliminary analysis on software worker behaviors reveals an alarming task dropping rate of 82.9%. These factors lead to the need for an automated recommendation system for CSD workers to improve the visibility and predictability of their success in the competition. To that end, this paper proposes a collaborative recommendation system for crowd workers. The proposed recommendation system method uses five input metrics based on workers' collaboration history in the pool, workers' preferences in taking tasks in terms of monetary prize and duration, workers' specialty, and workers' proficiency. The proposed method then recommends the most suitable tasks for a worker to compete on based on workers' probability of success in the task. Experimental results on 260 active crowd workers demonstrate that just following the top three success probabilities of task recommendations, workers can achieve success up to 86%

Razieh Saremi, Hamid Shamszare, Marzieh Lotfalian Saremi et al.

Software crowdsourcing platforms employ extrinsic rewards such as rating or ranking systems to motivate workers. Such rating systems are noisy and provide limited knowledge about workers' preferences and performance. To develop better understanding of worker reliability and trustworthiness in software crowdsourcing, this paper reports an empirical study conducted on more than one year's real-world data from TopCoder, one of the leading software crowdsourcing platforms. To do so, first, we create a bipartite network of active workers based on common task registrations. Then, we use the Clauset-Newman-Moore graph clustering algorithm to identify worker clusters in the network. Finally, we conduct an empirical evaluation to measure and analyze workers' behavior per identified community in the platform by workers' rating. More specifically, workers' behavior is analyzed based on their performances in terms of reliability, trustworthiness, and success; their preferences in terms of efficiency and elasticity; and strategies in terms of comfort, confidence, and deceitfulness. The main result of this study identified four communities of active workers: mixed-ranked, high-ranked, mid-ranked, and low-ranked. This study shows that the low-ranked community associates with the highest reliable workers with an average reliability of 25%, while the mixed-ranked community contains the most trustworthy workers with average trustworthiness of 16%. Such empirical evidence is beneficial to help exploring resourcing options while understanding the relations among unknown resources to improve task success.

Razieh Saremi, Hardik Yagnik, Julian Togelius et al.

The complexity of software tasks and the uncertainty of crowd developer behaviors make it challenging to plan crowdsourced software development (CSD) projects. In a competitive crowdsourcing marketplace, competition for shared worker resources from multiple simultaneously open tasks adds another layer of uncertainty to the potential outcomes of software crowdsourcing. These factors lead to the need for supporting CSD managers with automated scheduling to improve the visibility and predictability of crowdsourcing processes and outcomes. To that end, this paper proposes an evolutionary algorithm-based task scheduling method for crowdsourced software development. The proposed evolutionary scheduling method uses a multiobjective genetic algorithm to recommend an optimal task start date. The method uses three fitness functions, based on project duration, task similarity, and task failure prediction, respectively. The task failure fitness function uses a neural network to predict the probability of task failure with respect to a specific task start date. The proposed method then recommends the best tasks start dates for the project as a whole and each individual task so as to achieve the lowest project failure ratio. Experimental results on 4 projects demonstrate that the proposed method has the potential to reduce project duration by a factor of 33-78%.

Razieh Saremi, Marzieh Lotfalian Saremi, Sanam Jena et al.

Crowdsourcing is becoming an accepted method of software development for different phases in the production lifecycle. Ideally, mass parallel production through Crowdsourcing could be an option for rapid acquisition in software engineering by leveraging infinite worker resource on the internet. It is important to understand the patterns and strategies of decomposing and uploading parallel tasks to maintain a stable worker supply as well as a satisfactory task completion rate. This research report is an empirical analysis of the available tasks' lifecycle patterns in crowdsourcing. Following the waterfall model in Crowdsourced Software Development (CSD), this research identified four patterns for the sequence of task arrival per project: 1) Prior Cycle, 2) Current Cycle, 3) Orbit Cycle, and 4) Fresh Cycle.

Razieh Saremi, Ye Yang, Gregg Vesonder et al.

A typical crowdsourcing software development(CSD) marketplace consists of a list of software tasks as service demands and a pool of freelancer developers as service suppliers. Highly dynamic and competitive CSD market places may result in task failure due to unforeseen risks, such as increased competition over shared worker supply, or uncertainty associated with workers' experience and skills, and so on. To improve CSD effectiveness, it is essential to better understand and plan with respect to dynamic worker characteristics and risks associated with CSD processes. In this paper, we present a hybrid simulation model, CrowdSim, to forecast crowdsourcing task failure risk in competitive CSD platforms. CrowdSim is composed of three layered components: the macro-level reflects the overall crowdsourcing platform based on system dynamics,the meso-level represents the task life cycle based on discrete event simulation, and the micro-level models the crowd workers' decision-making processes based on agent-based simulation. CrowdSim is evaluated through three CSD decision scenarios to demonstrate its effectiveness, using a real-world historical dataset and the results demonstrate CrowdSim's potential in empowering crowdsourcing managers to explore crowdsourcing outcomes with respect to different task scheduling options.

Denisse Martinez Mejorado, Razieh Saremi, Ye Yang et al.

Context: The success of software crowdsourcing depends on steady tasks supply and active worker pool. Existing analysis reveals an average task failure ratio of 15.7% in software crowdsourcing market. Goal: The objective of this study is to empirically investigate patterns and effect of task diversity in software crowdsourcing platform in order to improve the success and efficiency of software crowdsourcing. Method: We propose a conceptual task diversity model, and develop an approach to measuring and analyzing task diversity.More specifically, this includes grouping similar tasks, ranking them based on their competition level and identifying the dominant attributes that distinguish among these levels, and then studying the impact of task diversity on task success and worker performance in crowdsourcing platform. The empirical study is conducted on more than one year's real-world data from TopCoder, the leading software crowdsourcing platform. Results: We identified that monetary prize and task complexity are the dominant attributes that differentiate among different competition levels. Based on these dominant attributes, we found three task diversity patterns (configurations) from workers behavior perspective: responsive to prize, responsive to prize and complexity and over responsive to prize. This study supports that1) responsive to prize configuration provides highest level of task density and workers' reliability in a platform; 2) responsive to prize and complexity configuration leads to attracting high level of trustworthy workers; 3) over responsive to prize configuration results in highest task stability and the lowest failure ratio in the platform for not high similar tasks.

Jordan Urbaczek, Razieh Saremi, Mostaan Lotfalian Saremi et al.

Context: Highly dynamic and competitive crowdsourcing software development (CSD) marketplaces may experience task failure due to unforeseen reasons, such as increased competition over shared supplier resources, or uncertainty associated with a dynamic worker supply. Existing analysis reveals an average task failure ratio of 15.7\% in software crowdsourcing markets. Goal: The objective of this study is to provide a task scheduling recommendation model for software crowdsourcing platforms in order to improve the success and efficiency of software crowdsourcing. Method: We propose a task scheduling method based on neural networks, and develop a system that can predict and analyze task failure probability upon arrival. More specifically, the model uses a range of input variables, including the number of open tasks in the platform, the average task similarity between arriving tasks and open tasks, the winner's monetary prize, and task duration, to predict the probability of task failure on the planned arrival date and two surplus days. This prediction will offer the recommended day associated with the lowest task failure probability to post the task. The proposed model is based on the workflow and data of Topcoder, one of the primary software crowdsourcing platforms. Results: We present a model that suggests the best recommended arrival dates for any task in the project with surplus of two days per task in the project. The model on average provided 4\% lower failure ratio per project.