Crowdsourcing Software Development: Task Assignment Using PDDL Artificial Intelligence Planning

1. Introduction

Crowdsourcing software development (CSD) uses an open call format online to catch a large number of workers to participate. The open call format of CSD involves three types of roles: the requester (i.e., for the one the project is undertaken), the platform (i.e., the service provider), and the CS developer (i.e., the person for coding and testing). This kind of call format always collects large numbers of self-selected tasks. In this process, several numbers of developers can register and select the task at the platform. The platform is also responsible to evaluate the submitted tasks to decide for the best solution, from the developers, to pay the rewards. The authors [1,2] stated that the selection of an effective and appropriate task from the extensive large set is a hectic work for CS developers. It is also a tiring and time-consuming job for platform workers to evaluate thousands of submitted tasks from developers. For example, [3] also maintained that finding an effective task from the submitted tasks is a hard and time-consuming job for the platform workers. Based on the studies by [4,5], receiving or assigning an improper task to improper CS developer may not only decrease the quality of the software deliverables but also causes overburden on both platform and developers. Mostly, workers view a fewer numbers of recent tasks are posted on the CS platform because hundereds of tasks are posted every day [6]. By considering the skills and expertise level of the CS developers, unrealistic matching of CS developers and tasks would affect the software quality. Synchronization between expertise or knowledge of developers and tasks is a serious problem which can benefit both (platform and CS developers) [7]. Based on the findings [8], CSD does not only face technical issues but also deals with human-related issues at the same time. It is an admitted statement that software is made for the people by the people [9]. The personality of CS developers is an important element of people factor which directly impacts the results or outcomes of task development in CSD [10]. Currently, the artificial intelligence (AI) methods being applied in a conventional system are growing for the solution of problems. For examples, these days, planning methods including robotics and space machine are widely used in some particular areas. Nowadays, researchers have also started to use these techniques in other areas [11,12]. This study is going to use the classical planning of AI techniques to assist the CSD platform and CS developers to assign and choose the task logically with their personality type match. By using this technique, this study not only provided the best solution to reduce the overburden of searching and matching the task but also increased the efficiency which helped in providing quality deliverables.

This paper comprises five sections. Section 1 describes the introduction of the study. In Section 2 related work and background information are discussed. In Section 3 the methodology and study setting are presented. Section 4 presents the analysis of the results and their discussion. The conclusion is given in Section 5.

2. Background and Related Work

Crowdsourcing is a distributed outsourcing to an indeterminate, usually outsized crowd of people in an open call format. It has attracted a great attention from the industry and academia. When employing crowdsourcing to accomplish software development tasks, CSD faces challenges of assigning, sorting, and searching the suitable developers for the specific tasks. Until today, most developed tasks are assigned in the form of bidding or competition. As a result, a large amount of human effort and time are wasted in the searching for the suitable task with personal preference. However, many CSD developers cannot compete for the tasks [5]. To recommend a CSD task to a developer [2] developed a content-based technique, which takes a record from the history of the developer i.e. winning and registration to match automatically the task and the developer. The authors [13] described and proposed bias correction in crowd data in the form of modeling. They used a gold standard data set to estimate the CSD workers model accuracy. However, this method was used in micro-tasking. [14] used an implicit modeling based on skills and interest of CSD worker to recommend the classification-based task. The authors [15] mentioned and proposed an approach based on task matching which encourages and motivates CSD workers to do a task for continuous and long run. This approach focused on the recommendation features of the tasks to be best matched with the workers.

The authors of [12] presented an architecture for general crowdsourced systems with an automated planner role. In this approach, two key differences were found. Firstly, random versus guided behavior was designed for assigning the CS workers. Secondly, the performance results of open CS were incomplete and poor. Instead, coordination algorithms which are currently used in guided crowdsourcing are not suitable for CSD for more useful solutions. The work in [11], suggested a guided crowdsourcing which tested the incorporate crowdsourcing. The algorithm was used to schedule the resources and dynamically assign micro-task to crowdsource workers. In that way, AI methods can evaluate to coordination with users for achieving the specific collective goal performance in a CSD environment. [8] presented a Planning Domain Definition Language (PDDL) approach to generalize CS environment. However, in this paper, we introduced AI planning by using PDDL in CSD especially in Competitive Scenario.

2.1 Use of Personality in Crowdsourcing Software Developing

In order to avoid the risks of giving the task to improper personality types of CSD, the authors [16] proposed a suitable model. According to that model, it was suggested that, a task assigned to a developer must be based on suitable personality types. For instance, the personality of a programmer should be Introvert (I), the personality of a system analyst should be Extrovert (E), and the tester has Sensing (S) and Thinking (T) personalities. The software designer should be with Intuitive (N) and Think (T) personalities. This is because the personality types of the developer are the important human aspects to ensure the quality of software tasks. It is also confirmed that a technically sound individual cannot perform satisfactorily unless he/she is assigned with development tasks based on the person's personality types. [17] mentioned that assigning a specific personality to the task in software development is well suited for their traits to increase the successful outcome of the tasks. In order to help participants with the detection of appropriate tasks according to their individual preferences, this study suggested new techniques to facilitate the required self-identification process. The classification of individual personality types was classified on the Myer Briggs Type Indicator (MBTI) test, which follows the combination of four-dimensional pairs. From the four dimensional pairs, there were 16 personality combinations as shown in Table 1. To evaluate the personality of a CSD worker, this study will use the MBTI personality type as the instrument. This instrument is widely used in software engineering the research [18-21].

3. Research Methodology

This study focused on searching for the possible results of the following key research questions by simulating the CSD developers and task:

(Research Question 1) Is it possible to generate plans to attract and assign the task to CSD developer who matches with the task and personality relationship criteria for the CSD tasks?

(Research Question 2) Is it possible to generate plans to lessen the overburden of platform and CSD developer for searching the open tasks and optimize the assigning and participation of CSD developer?

3.1 Implementation of the Planning

In order to get the answers of the above Research Questions, this study implemented the AI algorithm known as the automated planning to describe the basic mechanism and evaluation of CSD task assignments problem. This study used the PDDL, a planning language for AI planning systems. In PDDL first model, a domain explanation was defined, and then the problem which signified the situations will be defined. In CSD task assignment or sorting problem case, the domain description should have the defined objects, predicates, and actions which must relate with the tasks and CSD developer, as described in Tables 2–4. It is also very important that all criteria constraints and features are applicable to the execution of a well-defined plan.

3.2 Explanation of Predicates and Object Domain

When requester wants to resolve a task posted on the platform by a CSD developer through an open call format, the developer must possess the requirements to resolve the task and in the same way, the task poster needs to provide the basic for the solution. The platform is an agent, the initial state is the task on the platform, and the goal is to assign a task to requester that match the specific requirements. For improving the tasks assigning plan, a number of the premeditated elements are separated to the problem of crowdsourcing software development. The basic elements in competitive environment are uttered in Table 5.

3.3 Domain Actions Definition

To generate an intelligent plan for task matching, some actions needed to be defined in the algorithm.

• Match: Try to match a CSD developer to whom the task will be given by considering personality match.

• Assign: To assign a task to a matched personality if the task is posted according to his/her personality type.

• Submission: Mark the task as submitted when submitted from the CSD developer for competition

• Review: After submitting the task reviewed and evaluated for reward.

• Reward: If the submitted task wins the competition, then pays the reward.

3.4 Definition of the Problem

The context established for the task requirement in the matching situation presented by this study for a personality and a CSD task in terms of time and monetary benefits, are depicted in Fig. 1. As discussed in the problem section, it is possible to analyze and match a task for a CSD platform based on its firmed description. The CSD platform has the ability to match the CSD developers that are the best matched for the developing and solving of the task.

It is very essential to the relationship between the objects’ of personality and task with the conditioned personality, time, and monetary reward. This relationship sets a correlation between, incentive which a CSD developer attracts to participate (Get_reward ?p ?r), the personality type of the CSD developer that a CSD has (has_personlity ?P ?T ), if required, time constraints to compete the task (Task_ time ?t ?T)and submission of task (submission_for ?S ?T). To implement the plan and discover the arrangements and different situations between personalities and task (objects) problem files were formed. In order to incorporate all necessary set of objects for suggested combination, the initial state is where a set (T) is started. Tasks to be assigned, matched to a developer who required the set of the Personality (P). It is also mentioned that if the CSD task has an additional requirement such as special skills, time constraints, and incentive attraction, they have to motivate. To allocate a huge number of CSD developers to assign tasks on the CSD platform from the registered personalities who match the criteria is the goal and final state of the planner.

3.5 Generation of AI Plan

This section describes three situations where task and personality type are connected with predicates in a different context. The changing of two variables, i.e., reward (R) and time complexity (T) is the main goal. To represent some firm and normal situations in the domain of task matching, different results based on the merits and demerits of the classical planning approach are generated by each configuration. Table 6 represents the symbols used in predicates.

Situation 1: This situation was planned to illustrate an uncomplicated successful way, where each task was posted on the CSD platform with time constraints, the starting and ending date, monetary incentive, and registration with MBTI Personality test. At least one matching personality with the criteria was available. In the CSD context, this situation was simply matched by a relationship where enough CSD developers were available in the competition to match all tasks conditions; with no adjustment, was required for the variables into the system.

P1 has R and has t to S1, S2 the T1, T3 for competition of r

P2 has R and has t to S3, S1 the T2, T1 for competition of r

P3 has R and has t to S4, S5 the T4, T5 for competition r

PN has R and t to Sn the Tn for competition of r

A CSD developer with matching variables for each task was preferred for the generated plan.

Situation 2: In this situation, no matching personality type was available, for a single task within the registered crowd. It was also mentioned that special skills required to match the task and reward were also put high, but there is no CSD developer willing to register itself for the reason of time complexity.

P1 has R and is available for t, take T1, and S for r

P5 has R, and available for t, take T1 and S for r

Pn not R, and not available for t, do not take T1 and S for r

For this situation, there was no plan generated as a result. As the goal was to assign the tasks, it was impossible find the suitable matching for the task due to time complexity factor. When analyzing the requester for the set of tasks where there was no matching in the CSD crowd the algorithm did not generate any plan. In this situation, in order to achieve the goal successfully, this study suggested that the variables should be adjusted according to the situation to better match the task and variables. Moreover, the complexity was relaxed for (Tn).

Situation 3: The aim of this situation was, to adjust the time complexity when the task did not match in the previous situation. The time of submission of task reset in order to attract more CSD developers for competition. In this situation, the plan was generated as a result. The features of this algorithm allowed changing variables according to the different situations, for instance, to increase the monetary reward and reduce time complexity. Someone can also increase the time complexity factor and reduce the rewards incentive. The plans will be generated if there is flexibility in variables and at least one match is available, which matches the task and personality relationship.

4. Results and Discussion

(Research Question 1) Is it possible to generate plans to attract and assign the task to CSD developer who matches with the task and personality relationship criteria for the CSD tasks?

When the requirement for each task is matched with at least one CSD developer the algorithm generates plans. It was also mentioned that whenever there is no requirement matched with the task, the algorithm did not generate the plan in that situation. It is quite obvious that AI planning technique is always searching for a CSD worker who meets all the requirement or at least one required conditions for matching with the task and personality relationship to ensure the deliverable quality for submission.

(Research Question 2) Is it possible to generate the plans to lessen the burden of platform and CSD developer for searching the open tasks and optimizing the assigning, and participation of CSD developer?

Within the competitive environment, considering only the classical planning approach, the results for this algorithm showed that to generate plans, can be impossible when you optimize any variable. This occurs regardless of whether their feasibility is suitable or not. This approach indicated that the algorithm is flexible enough to test many configurations, in different contexts in order to best match with task and personality types.

In this context, a new research question has been determined throughout the research. Can the resultant plans help to optimize the progress in the quality of the solution required by the requester? The answer to this research question is that by the setting of situations and different variables this algorithm is generating better plans, as discussed in Situation 3 above. The requirements of variables should be flexible in order to achieve the optimized results. For instance, to increase time complexity, short duration, reward motivation, and relaxing, special skills are required. Assigning the right task to the right CSD developer according to his/her personality types and getting benefits and exploring more of AI Planning techniques in crowdsourcing software development is at the initial state.

5. Conclusion

As discussed above, the main objective of this research is to evaluate a PDDL planning for providing a relationship to match a CSD developer with personality and task. Moreover, it also provides a quality submission and reduces the burden from the platform as well as the CSD developer to match the task and CSD developer. From the results, it was very obvious that the automated planning has a significant impact, as the planner has to pre-categorize the tasks and personality types. It was also noted that planning algorithm has the capacity of allowing changes in the variables and it provided more and proper match.

The limitation of this planning algorithm is that in inflexible conditions for the matching of CSD task and developer, the plans will not be generated. Therefore, the variables must be adjusted according to the situations to achieve the best results.

It is suggested that CSD platforms may adopt the automated planning by using PDDL AI techniques, in order to reduce a large amount of human effort wasted in searching and assigning the tasks. In future, we will implement this algorithm in different situations and on real crowdsourcing platforms to get more validity.

Acknowledgement

This work is supported in part by the National Key Research and Development Program (No. 2016- YFB-0502004), the National Natural Science Foundation of China (No. 61374214), the and the Open Project of Beijing Key Laboratory of Mobile Computing and Pervasive Devices.