Abstract1 The DAG scheduling problem is a rich land of research and a plethora of algorithms for solving this problem have been reported in the literature . However , designing a scheduling algorithm of low complexity without sacrificing performance remains a challenging obstacle from a practical perspective . In this paper , we present a local search-based scheduling algorithm that attempts to meet this challenge . The proposed algorithm is called Fast Assignment using Search Technique -LRB- FAST -RRB- . Its overall time complexity is only where e is the number of edges in the DAG . The algorithm works by first generating an initial solution and then refining it using local neighborhood search . The algorithm outperforms numerous previous algorithms while taking dramatically smaller execution times . The distinctive feature of our research is that the performance evaluation is not carried out using simulation , rather we have tested our proposed algorithm and compared it with other algorithms using a parallel compiler with real applications on the Intel Paragon . 1 Introduction To efficiently exploit the tremendous potential of highperformance architectures , the tasks of a parallel application must be carefully decomposed and scheduled to the processors so that the execution time is minimized . When the characteristics of the parallel program , such as execution times of the tasks , amount of communication data , and task dependencies are known a priori , scheduling can be done statically . The parallel program , with such known information , can be modeled as a nodeand edge-weighted directed acyclic graph -LRB- DAG -RRB- , in which the nodes and edges represent tasks and messages , respectively . With such a static model , the scheduler is invoked off-line or during compile-time and thus can afford moderate time complexity in order to generate a better schedule . This form of multiprocessor scheduling problem is called static scheduling or DAG scheduling . Static scheduling in most cases is NP-complete -LSB- 3 -RSB- , -LSB- 4 -RSB- , and optimal solutions exist only in three simple cases : -LRB- i -RRB- scheduling a tree-structured DAG with identical node weights to an arbitrary number of processors -LSB- 3 -RSB- , -LRB- ii -RRB- scheduling an arbitrary DAG with identical node weights to two processors -LSB- 3 -RSB- , and -LRB- iii -RRB- scheduling an intervalordered DAG to an arbitrary number of processors -LSB- 3 -RSB- . Thus , heuristic approaches are sought to tackle the 1 . This research was supported by the Hong Kong Research Grants Council under contract number HKUST 619/94E . O e -LRB- -RRB- problem under more realistic cases . There have been a large number of static scheduling heuristics reported in the literature -LSB- 1 -RSB- , -LSB- 9 -RSB- , -LSB- 11 -RSB- , -LSB- 13 -RSB- . However , while generating good solutions , very few of them have a low complexity . Thus , most of the algorithms are impractical in a real environment . In a recent study -LSB- 1 -RSB- , we compared 21 such algorithms and made a number of findings . For example , we observed that an scheduling algorithm -LRB- here v denotes the number of nodes in the DAG -RRB- can take more than an hour to schedule a DAG with 1,000 nodes -LRB- typical in many applications -RRB- . Taking such a large amount of time to produce a schedule for an application is a major obstacle in using these algorithms with parallel compilers . Some algorithms have low complexities but their solution quality is not satisfactory -LSB- 1 -RSB- . The objective of this study is to propose a low complexity DAG scheduling algorithm that can produce efficient schedules . The most common technique for DAG scheduling is to assign priorities to the nodes of the DAG , and allocate the higher priority nodes to the appropriate time slots on the processors -LSB- 7 -RSB- . The solution quality -LRB- schedule length -RRB- is highly dependent on the accuracy of the priorities . To determine the priorities more accurately , some algorithms spend extra computation steps . Backtracking or recalculation of priorities can incur even higher complexity . While it is understood that static scheduling is off-line and some extra time can be afforded in generating a better solution , the time complexity of an algorithm is an important issue from a practical point of view . In this regard , some of the pioneering work done by Yang and Gerasoulis -LSB- 18 -RSB- addressed this problem and proposed some novel techniques for reducing the time complexity of the scheduling algorithms . The objective of our work is to propose an algorithm that has a comparable or lower complexity while producing even better solutions . We propose a new algorithm using a technique called local search -LSB- 6 -RSB- , -LSB- 15 -RSB- which has been successfully applied to solve many NP-hard optimization problems -LSB- 12 -RSB- . Our algorithm has two phases . In the first phase , we generate a moderately optimized schedule quickly without spending a large amount of time to generate a good schedule at one shot . Then , in the second phase , we employ the local search technique to refine the schedule . The overall time complexity of our algorithm , which is called Fast Assignment using Search Technique -LRB- FAST -RRB- , is only where e is the number of edges in the DAG . The performance evaluation is carried out not only using simulation for randomly generated task graphs , but the proposed algorithm is also tested and compared it with O v3 -LRB- -RRB- O e -LRB- -RRB- FAST : A Low-Complexity Algorithm for Efficient Scheduling of DAGs on Parallel Processors Yu-Kwong Kwok , Ishfaq Ahmad , and Jun Gu Department of Computer Science The Hong Kong University of Science and Technology , Hong Kong . other algorithms using an actual parallel compiler with real applications on the Intel Paragon . The rest of this paper is organized as follows . In the next section , we introduce the parallel program model used in the DAG scheduling problem . In Section 3 , we describe some of the related work on DAG scheduling reported in the literature . In Section 4 , we discuss our proposed algorithm . Section 5 contains the performance results , and the final section concludes the paper . 2 Background In the static scheduling problem , a parallel program is typically modeled by a directed acyclic graph -LRB- DAG -RRB- , where V is a set of v nodes and E is a set of e directed edges . A node in the DAG represents a task which is a set of instructions that must be executed sequentially in the same processor . The weight on a node is called the computation cost of a node and is denoted by . The edges in the DAG , each of which is denoted by , correspond to the communication messages and precedence constraints among the nodes . The weight on an edge is called the communication cost of the edge and is denoted by . The source node of an edge is called the parent node while the destination node is called the child node . A node with no parent is called an entry node and a node with no child is called an exit node . A Critical Path -LRB- CP -RRB- of a DAG is a set of nodes and edges constituting a path which has the largest length . The length of the CP is the sum of the computation costs and communication costs along the path . The communication-to-computation-ratio -LRB- CCR -RRB- of a parallel program is defined as its average communication cost divided by its average computation cost on a given system . A node is not ready -LRB- i.e. , can not start execution -RRB- before it gathers all of the messages from its parent nodes . The communication cost among two nodes assigned to the same processor is assumed to be zero . If node is scheduled to processor P , and denote the start time and finish time of on processor P , respectively . After all nodes have been scheduled , the overall execution time or schedule length , is defined as across all processors . The objective of a DAG scheduling algorithm is to assign and schedule the nodes to processors such that the schedule length is minimized without violating the precedence constraints . The CP provides a lower bound on the schedule length of the DAG . Thus , nodes on the CP must be given higher priorities . To identify a CP node -LRB- CPN -RRB- , two attributes , the t-level and b-level -LSB- 18 -RSB- , can be used . The t-level of a node is the length of the longest path from an entry node to excluding . The b-level of a node is the length of the longest path from to an exit node . The CPNs of a DAG are the nodes with the largest sum of t-level and blevel . Since the computations of t-level and b-level take time , the CPNs can be identified also in time . For scheduling , the t-level and b-level of a node can be used in a variety of ways . Some algorithms use t-level as a G V E , -LRB- -RRB- = ni w n i -LRB- -RRB- ni n j , -LRB- -RRB- c n i nj , -LRB- -RRB- ni ST n i P , -LRB- -RRB- FT n i P , -LRB- -RRB- ni maxi FT n i P , -LRB- -RRB- -LCB- -RCB- n in i w ni -LRB- -RRB- n i ni O e -LRB- -RRB- O e -LRB- -RRB- node priority , while some algorithms use b-level . Some algorithms use a variant of b-level , called the static b-level or simply static level -LRB- SL -RRB- , in which only the computation costs and not the communication costs are taken into account . There are also two additional attributes called the as-soon-as-possible -LRB- ASAP -RRB- start time and as-late-aspossible -LRB- ALAP -RRB- start time -LSB- 17 -RSB- . The ASAP value is just the t-level . The ALAP start time is the largest possible start time of a node bounded by the CP length . Thus , the ALAP value is equal to the CP length minus the b-level of the node . Given the ASAP and ALAP values , the CPNs , whose ASAP and ALAP values are equal , can be easily identified . Notice that the CP can change during the scheduling process because some of the edges are zeroed if the connected nodes are scheduled to the same processors . To overcome this situation , some algorithms re-compute the values of t-level and b-level . This increases the complexity since , if the scheduling algorithm takes v steps to schedule all the nodes , the re-computation of priorities can take an overall time of , which can be in the worst case . In Figure 1 , an example DAG and the values of SL , tlevel -LRB- ASAP -RRB- , b-level , and ALAP of its nodes are provided . The CPNs of the DAG are shown in dark and the edges on the CP are shown with thick arrows . O ev -LRB- -RRB- O v 3 -LRB- -RRB- node SL t-level b-level ALAP * 12 37 8 6 23 14 8 3 23 14 9 3 20 17 10 3 30 7 5 10 15 22 * 5 22 15 22 5 18 15 22 * 1 36 1 36 n1 n2 n3 n4 n5 n6n7 n8n9 -LRB- a -RRB- -LRB- b -RRB- Figure 1 : -LRB- a -RRB- A task graph ; -LRB- b -RRB- The static levels -LRB- SLs -RRB- , t-levels -LRB- ASAP times -RRB- , b-levels , and ALAP times of the nodes -LRB- CPNs are marked by an asterisk -RRB- . n1 2 n2 3 n5 5 n 91 n3 3 n7 4 n4 4 n8 4 n6 4 1 20 1 1 1 1 4 5 1 5 5 10 10 10 10 3 Related Work A large number of DAG scheduling algorithms have been reported in the literature . In this section , we describe four related scheduling algorithms : the Mobility Directed -LRB- MD -RRB- algorithm -LSB- 17 -RSB- , the Earliest Task First -LRB- ETF -RRB- algorithm -LSB- 8 -RSB- , the Dynamic Level Scheduling -LRB- DLS -RRB- algorithm -LSB- 14 -RSB- , and the Dominant Sequence Clustering -LRB- DSC -RRB- algorithm -LSB- 18 -RSB- . 3.1 The MD Algorithm The MD -LRB- Mobility Directed -RRB- algorithm selects a node ni for scheduling based on an attribute called the relative mobility , which is defined as dividing the difference between the ALAP and ASAP by . At each step , the MD algorithm selects the node with the smallest relative mobility for scheduling . In finding a processor for the node , the MD algorithm schedules the node to the first processor that can accommodate the node . The time complexity of the MD algorithm is . 3.2 The ETF Algorithm The ETF -LRB- Earliest Time First -RRB- algorithm computes , at each step , the earliest start times for all ready nodes and then selects the one with the smallest start time . When two nodes have the same value of their earliest start times , the ETF algorithm breaks the tie by scheduling the one with the higher static level . The time complexity of the ETF algorithm is . 3.3 The DLS Algorithm The DLS -LRB- Dynamic Level Scheduling -RRB- algorithm uses an attribute called dynamic level -LRB- DL -RRB- which is the difference between the static b-level of a node and its earliest start time -LRB- i.e. , t-level -RRB- on a processor . The nodeprocessor pair that gives the largest value of DL is selected for scheduling . The time complexity of the DLS algorithm is . The high complexity of the algorithm is due to the pair-wise matching of the nodes to processors . 3.4 The DSC Algorithm The DSC -LRB- Dominant Sequence Clustering -RRB- algorithm considers the Dominant Sequence -LRB- DS -RRB- of a graph . The DS is simply the CP of the partially scheduled DAG . The algorithm is briefly described below . The DSC algorithm tracks the CP of the partially scheduled DAG at each step by using the composite attribute -LRB- b-level + t-level -RRB- as the priority of a node . The DSC algorithm does not select the node with the highest priority for scheduling unless the node is ready . This is done in order to lower the time complexity of the algorithm because the t-level of a node can be computed incrementally and the b-level does not change until the node is scheduled . The time complexity of the DSC algorithm is . 4 The Proposed Algorithm The local search-based algorithm called Fast Assignment using Search Technique -LRB- FAST -RRB- , has two phases : 1 -RRB- generate an initial schedule using the classical list scheduling method in time using the given w ni -LRB- -RRB- O v 3 -LRB- -RRB- O pv 2 -LRB- -RRB- O pev -LRB- -RRB- O e v + -LRB- -RRB- vl og -LRB- -RRB- O e -LRB- -RRB- number of available processors ; 2 -RRB- refine the schedule using local neighborhood search in time . In the following , we first describe how to determine node priorities and generate an initial schedule in time . Then we describe how the local search technique can be applied to improve upon the initial schedule . 4.1 Determining Node Priorities To generate an initial schedule , we employ the traditional list scheduling approach ? construct a list and schedule the nodes on the list one by one to the processors . The list is constructed by ordering the nodes according to the node priorities . The list is static so that the order of nodes on the list will not change during the scheduling process . The reason is that as the objective of our algorithm is to produce a good schedule in time , we do not re-compute the node priorities after each scheduling step while generating the initial schedule . Nonetheless , if the schedule length of the initial schedule is optimized , the subsequent local search process can start at a better solution point and thereby , can generate a better final schedule . In order to construct a static list of nodes ordered in accurate priorities , we employ a novel method for the arrangement of nodes . As discussed earlier , the CP nodes -LRB- CPNs -RRB- are the more important nodes because their scheduling order can have a direct impact on the schedule length . Thus , we assign the highest priority to CPNs . However , a CPN can not be scheduled before all of its parent nodes are scheduled . To tackle this problem , we partition the nodes of the DAG as follows : In a connected graph , an In-Branch Node -LRB- IBN -RRB- is a node , which is not a CPN , and from which there is a path reaching a Critical Path Node -LRB- CPN -RRB- . An Out-Branch Node -LRB- OBN -RRB- is a node , which is neither a CPN nor an IBN . For the DAG shown earlier in Figure 1 , the CPNs are shown with dark color , the IBNs are uncolored . There is no OBN in this DAG . When a CPN is considered for scheduling , all the IBNs reaching it as well as its parent CPN must have been scheduled . Thus , the IBNs reaching a CPN must occupy higher positions on the list than the CPN itself . The OBNs are relatively less important nodes so that they can occupy lower positions on the list . To further differentiate the importance among the IBNs , we order the IBNs in a decreasing order of b-levels . The OBNs are also ordered in increasing b-levels . Given these observations , we can formalize our method of constructing the scheduling list , which is called the CPN-Dominate List , as follow . CPN-Dominate List : -LRB- 1 -RRB- Make the entry CPN to be the first node in the list . Set Post to 2 . Let be the next CPN . Repeat -LRB- 2 -RRB- If has all its parent nodes in the list then -LRB- 3 -RRB- Put at Post in the list and increment Post . -LRB- 4 -RRB- else -LRB- 5 -RRB- Suppose is the parent node of which is not in the list and has the largest b-level . Ties are broken by O e -LRB- -RRB- O e -LRB- -RRB- O e -LRB- -RRB- n x nx nx ny nx choosing the parent node with a smaller t-level . If has all its parent nodes in the list , put at Post in the list and increment Post . Otherwise , recursively include all the ancestor nodes of in the list so that the nodes with larger b-levels are considered first . -LRB- 6 -RRB- Repeat step -LRB- 5 -RRB- until all the parent nodes of are in the list . Put in the list at Post . -LRB- 7 -RRB- endif -LRB- 8 -RRB- Make to be the next CPN . Until all CPNs are in the list . -LRB- 9 -RRB- Append all the OBNs to the list in a decreasing order of b-level . It is obvious that the above procedure for constructing the CPN-Dominate List takes only time since each edge is visited once . 4.2 The Initial Schedule Using the CPN-Dominate List , we can schedule the nodes on the list one after another to the processors . Again , in order not to incur high complexity , we do not search for the earliest slot on a processor . Instead , we simply schedule a node to the ready-time of a processor . Initially , the ready-time of all available processors are zero . After a node is scheduled to a processor , the ready-time of that processor is updated . By doing so , a node is scheduled to a processor that allows the earliest start time , which is determined by checking the processor?s ready-time with the node?s data arrival time -LRB- DAT -RRB- . The DAT of a node can be computed by taking the maximum value among the message arrival times across the its parent nodes . If the parent is scheduled to the same processor as the node , the message arrival time is simply the parent?s finish time ; otherwise it is equal to the parent?s finish time -LRB- on a remote processor -RRB- plus the communication cost of the edge . Not all processors need to be checked in this process . Instead , we can examine the processors accommodating the parent nodes together with a new processor -LRB- if any -RRB- . The procedure for generating the initial schedule can be formalized below . InitialSchedule -LRB- -RRB- -LRB- 1 -RRB- Construct the CPN-Dominate List . Repeat -LRB- 2 -RRB- Remove the first node from the list . -LRB- 3 -RRB- Schedule to the processor , among the processors accommodating the parent nodes of together with a new processor -LRB- if any -RRB- , that allows the earliest start time by checking ? s DAT with the ready-times of the processors . Until the list is empty . The time complexity of InitialSchedule -LRB- -RRB- is determined as follows . The first step takes time . In the repeat loop , the dominant step is the procedure to determine the data arrival time of a node . The cumulative time complexity of this step throughout the execution of the repeat loop is also . Thus , the time complexity of InitialSchedule -LRB- -RRB- is again . To illustrate how InitialSchedule -LRB- -RRB- works , we can consider the DAG shown in Figure 1 -LRB- a -RRB- again . The CPN - Dominate List is . Note ny ny ny nx nx nx O e -LRB- -RRB- n i n i ni n i O e -LRB- -RRB- O e -LRB- -RRB- O e -LRB- -RRB- n1 n3 n 2 n , 7 n 6 n , 5 n4 n8 n9 , , , , , , -LCB- -RCB- that is considered after because has a smaller tlevel . Using the CPN-Dominate List , we can consider the initial schedule produced by InitialSchedule -LRB- -RRB- . For comparison , the schedules generated by the MD , ETF , and DLS algorithms are shown in Figure 2 . The schedule generated by the DSC algorithm is shown in Figure 3 . The schedule generated by InitialSchedule -LRB- -RRB- is shown in Figure 4 -LRB- a -RRB- . Note that the ETF and DLS algorithms generate the same schedule . As can be seen , the MD algorithm produces the worst schedule . This is due to the fact that the MD algorithm does not schedule a node to the earliest possible time slots even though it re-computes priorities at each step . The MD algorithm schedules the node late because it schedules the node too early so that it blocks . The schedule generated by the ETF and DLS algorithm is better but is still not satisfactory . The problem is that they schedule the node early because it has a higher value of static level -LRB- SL -RRB- . But is in fact not as important as which should have occupied an earlier slot . As a result , also starts late and the schedule length can not be reduced . The DSC algorithm generates a slightly better schedule . The problem with the schedule is that the node is scheduled to a late time slot because its parent is not scheduled to the same processor due to the minimization of ? s start time . The schedule length of generated by InitialSchedule -LRB- -RRB- is the shortest even though it does not employ sophisticated -LRB- and hence time n 8 n6 n6 PE 0 PE 1 PE 1PE 0 PE 1 n1 2 n2 3 n5 5 n91 n3 3 n7 4 n4 4 5 10 15 20 n8 4 n6 4 Figure 2 : Schedule generated by -LRB- a -RRB- the MD algorithm -LRB- schedule length = 32 -RRB- ; -LRB- b -RRB- the ETF and DLS algorithm -LRB- schedule length = 29 -RRB- . 25 30 32 n12 n2 3 n5 5 n91 n3 3 n7 4 n4 4 5 10 15 20 n8 4 n6 4 25 30 n4 n5 n4 n5 n5 n2 n7 n8 n4 n4 consuming -RRB- strategies to compute priorities and to select the best time slots for nodes . However , the initial schedule can be further refined to obtain a better solution , as will be discussed below . PE 0 PE 1 PE 1 PE 1 n12 n2 3 n5 5 n91 n3 3 n7 4 n4 4 5 10 15 20 n8 4 n6 4 25 30 Figure 3 : Schedule generated by the DSC algorithm -LRB- schedule length = 27 -RRB- . PE 0 PE 1 PE 2 PE 3 n1 2 n2 3 n5 5 n91 n3 3 n7 4 n4 4 n8 4 n6 4 5 10 15 20 24 Figure 4 : -LRB- a -RRB- Schedules generated by the InitialSchedule -LRB- -RRB- -LRB- schedule length = 24 -RRB- ; -LRB- b -RRB- The final schedule after the local search process with node n6 is transferred to PE 3 -LRB- schedule length = 23 -RRB- . PE 0 PE 1 PE 3 PE 3 n1 2 n2 3 n5 5 n91 n3 3 n7 4 n4 4 n8 4 n6 4 5 10 15 20 24 4.3 Local Search Local search is an old technique for combinatorial optimization . It has been applied to solve NP-hard optimization problems -LSB- 6 -RSB- . The principle of local search is to refine a given initial solution point in the solution space by searching through the neighborhood of the solution point . Recently a number of efficient heuristics for local search , i.e. , conflict minimization random selection / assignment , and pre - and partial selection/assignment , have been developed -LSB- 5 -RSB- , -LSB- 15 -RSB- , -LSB- 16 -RSB- . Sosic and Gu developed four efficient local search algorithms for a benchmark problem for constrained optimization problems , i.e. , the n-queen problem . The Queen_Search1 -LRB- QS1 -RRB- algorithm is a probabilistic local search algorithm that runs in approximately O -LRB- nlogn -RRB- time . The QS2 and QS3 are near linear local search algorithms with random selection/assignment and partial selection/assignment -LSB- 15 -RSB- . The QS4 algorithms is a linear time local search algorithm with pre - and partial random selection/assignment -LSB- 16 -RSB- . To apply the local search technique to the DAG scheduling problem , we have to define a neighborhood of the initial solution point -LRB- i.e. , the initial schedule -RRB- . A simple neighborhood point of a schedule in the solution space is another schedule which is obtained by transferring a node from a processor to another processor . In the DAG scheduling problem , one method to improve the schedule length is to transfer a blocking node from a processor to another processor . The notion of blocking is simple : a node is called blocking if removing it from its original processor can make the succeeding nodes to start earlier . In particular , we are interested in transferring the nodes that block the CPNs because the CPNs are the more important nodes . However , high complexity will result if we attempt to locate the actual blocking nodes on all the processors . Thus , in our approach , we only generate a list of potential blocking nodes which are the nodes that may block the CPNs . Again , to maintain complexity low , the blocking node list is static and is constructed before the search process starts . A natural choice of blocking node list is the set of IBNs and OBNs because these nodes have the potential to block the CPNs in the processors . In the schedule refinement phase , the blocking node list defines the neighborhood that the local search process will explore . To illustrate how the search process works , we can consider the initial schedule of the example DAG discussed previously . The blocking node list of the DAG is . We can notice that the node blocks the CPN . In the local search process , it is highly probable that is selected for transferring . Suppose it is transferred from PE 1 to PE 3 . The resulting schedule is shown in Figure 4 -LRB- b -RRB- . We can notice that even though the start times of the and are increased the final schedule length is shorten . 4.4 The Search-Based Scheduling Based on the search process mechanisms and the n 2 n , 3 n4 n5 n6 n8 , , , , -LCB- -RCB- n6 n9 n6 n5 n8 neighborhood for searching , we can formalize the proposed FAST -LRB- Fast Assignment using Search Technique -RRB- algorithm below . The FAST Algorithm : -LRB- 1 -RRB- InitialSchedule -LRB- -RRB- -LRB- 2 -RRB- Construct the blocking node list which contains all the IBNs and OBNs . -LRB- 3 -RRB- searchstep = 0 ; -LRB- 4 -RRB- do -LCB- / * search * / -LRB- 5 -RRB- Pick a node randomly from the blocking node list . -LRB- 6 -RRB- Pick a processor P randomly . -LRB- 7 -RRB- Transfer to P. -LRB- 8 -RRB- If schedule length does not improve , transfer back to its original processor . -LRB- 9 -RRB- -RCB- while -LRB- searchstep + + < MAXSTEP -RRB- ; The time complexity of the FAST algorithm is determined as follow . As discussed earlier , the procedure InitialSchedule -LRB- -RRB- takes time . The blocking node list can be constructed in time as the IBNs and OBNs are already identified in the procedure InitialSchedule -LRB- -RRB- . In the main loop , the node transferring step takes time since we have to re-visit all the edges once after transferred the node to a processor in the worst case . For our experiments , the constant MAXSTEP can be as small as 100 even for huge DAGs with tens of thousands of nodes . Indeed , for the results to be presented in the next section , the value of MAXSTEP is fixed at 64 . Thus , the overall complexity of the FAST algorithm is . 5 Performance In this section , we present the performance results of the FAST algorithm and also compare them with those of DSC , MD , ETF , and DLS algorithms . We performed experiments using real workload generated from serial applications by a prototype parallelization and scheduling tool called CASCH -LRB- Computer Aided SCHeduling -RRB- -LSB- 2 -RSB- . The CASCH tool generates a task graphs from a sequential program , uses a scheduling algorithm to perform scheduling , and then generates the parallel code in a scheduled form for the Intel Paragon . The timings for the nodes and edges on the DAG are assigned through a timing database that was obtained through benchmarking . CASCH also provides a graphical interface to interactively run and test various algorithms including the ones discussed in this paper . Instead of just measuring the schedule length through a Gantt chart , we measure the running time of the scheduled code on the Paragon . Various scheduling algorithms , therefore , can be more accurately tested and compared through CASCH using real applications on an actual machine . We also performed additional experiments with randomly generated large DAGs consisting of thousands of nodes . Notice that the MD and DSC algorithms assume unlimited number of processors while others do not . Thus , to investigate the performance of the algorithms in a fair manner , we give more than enough processors to all the algorithms . ni n i n i O e -LRB- -RRB- O v -LRB- -RRB- O e -LRB- -RRB- O e -LRB- -RRB- 5.1 Real Workload In our first experiment , we tested the FAST algorithm with the DAGs generated from three real applications : Gaussian elimination , Laplace equation solver and Fast Fourier Transform -LRB- FFT -RRB- -LSB- 2 -RSB- , -LSB- 10 -RSB- , -LSB- 17 -RSB- . The Gaussian elimination and Laplace equation solver applications operate on matrices . Thus , the number of nodes in the DAGs generated from these applications are related to the matrix dimension N and is about . On the other hand , the FFT application accepts the number of points as input . We examine the performance in three aspects : application execution time , number of processors used and the scheduling algorithm running time . The results for the Gaussian elimination are shown in Figure 5 . In the table shown in Figure 5 -LRB- a -RRB- , we normalized the application execution times obtained through all the algorithms with respect to those obtained through the FAST algorithm . We can notice that the programs scheduled by the FAST algorithm are 3 % to 15 % faster than the other algorithms . Note that the results of the DSC algorithm for matrix dimensions 16 and 32 were not available because the DSC used more than the available Paragon processors in scheduling the parallel program . In general , the DSC algorithm uses processors . Concerning the number of processors used , the FAST , ETF and DLS algorithms used about the same amount of processors . The number of processors used by all the algorithms is shown in Figure 5 -LRB- b -RRB- . The scheduling times of all the algorithms are shown in Figure 5 -LRB- c -RRB- . As can be seen , the DSC algorithm is the fastest algorithm and the proposed FAST algorithm is very close to it . The ETF and DLS algorithms running times are relatively large . The MD algorithm is much slower than the other algorithms . Indeed , the MD algorithm is about times slower than the other algorithms . The results for the Laplace equation solver are shown in Figure 6 . The percentage improvements of the FAST algorithm over the other algorithms are up to 25 % . The number of processors used in tis case are about the same for the FAST , MD , ETF and DLS algorithms . The DSC algorithm again uses more processors than the other algorithms . For the scheduling times , the FAST algorithm is the fastest among all the algorithms . The MD algorithm is again times slower than the other algorithms . The results for the FFT are shown in Figure 7 . The FAST algorithm is again better than all the other four algorithms in terms of execution time and running times . 5.2 Random DAGs To test the scalability and robustness of the FAST algorithm , we performed experiments with very large random DAGs . These DAGs were synthetically generated in the following manner . Given the size of the DAG -LRB- i.e. , v -RRB- , we first randomly generated the height of the DAG from a uniform distribution with mean roughly equal to . For each level , we generated a random number of nodes which was also selected from a uniform distribution with mean roughly equal to . Then , we connected the O N 2 -LRB- -RRB- O v -LRB- -RRB- O v -LRB- -RRB- O v -LRB- -RRB- v v nodes from the higher level to lower level randomly . The edge weights were also randomly generated . As the real workload discussed above consists of mainly sparse DAGs , the random DAGs generated were deliberately made denser . The sizes of the random DAGs were varied from 2000 to 5000 with an increment of 1000 . For these graphs , we simply measured the schedule length produced by an algorithm . The results for the random DAGs are shown in Figure 8 . Note that the MD algorithm was excluded from the comparison because it took more than 8 hours to produce a schedule for a 2000-node DAG . The FAST algorithm performed slightly worse the ETF and DLS algorithm but still outperformed the DSC algorithm by 7 % to 12 % margin . From Figure 8 -LRB- b -RRB- , we note that the DSC algorithm used an unrealistic number of processors . Concerning the scheduling times , we can immediately note from Figure 8 -LRB- c -RRB- that the ETF and DLS algorithms were considerably slower than the FAST and DSC algorithms . The proposed FAST algorithm outperforms the DSC algorithm both in terms of solution quality and complexity . It also outperforms the ETF , MD , and DLS algorithms most of the time . Furthermore , the ETF , MD , and DLS algorithms need a huge amount of scheduling time to produce a schedule for a realistically large DAG -LRB- note that the DAG for the Gaussian elimination application with matrix dimension 128 has 8192 nodes -RRB- . Thus , the FAST FAST DSC MD ETF DLS 1.00 1.00 1.00 1.00 1.05 1.08 N.A. N.A. 1.00 1.03 1.08 1.10 1.00 1.07 1.10 1.15 1.00 1.08 1.10 1.14 4 8 16 32 Algorithm -LRB- a -RRB- Normalized execution times of Gaussian elimination on the Intel Paragon . -LRB- b -RRB- Number of Processors used for the Gaussian elimination . FAST DSC MD ETF DLS 0.06 0.09 0.15 0.52 0.04 0.06 0.09 0.21 6.33 6.85 39.54 266.89 0.02 0.06 0.24 2.41 0.08 0.09 0.42 4.00 4 -LRB- 20 -RRB- 8 -LRB- 54 -RRB- 16 -LRB- 170 -RRB- 32 -LRB- 594 -RRB- Algorithm Matrix Dimension -LRB- Number of Tasks -RRB- -LRB- c -RRB- Scheduling times -LRB- sec -RRB- on a SPARC Station 2 for the Gaussian elimination . Figure 5 : Normalized execution times , number of processors used , and scheduling algorithm running times for the Gaussian elimination for all the scheduling algorithms . Matrix Dimension FAST DSC MD ETF DLS 4 8 16 32 5 22 95 128 2 3 4 7 3 7 16 32 3 7 16 32 4 8 16 32 Algorithm Matrix Dimension algorithm is more preferable for practical use in parallel program scheduling . 6 Conclusions The algorithm complexity and quality of the solution can become conflicting goals in the design of efficient scheduling algorithms . Our study indicates that not only does the solution qualities of existing algorithms differ considerably but their running times can vary by even huge margins . In this paper , we have presented a new scheduling algorithm which takes linear time to generate solutions with high quality . We have compared the algorithm with a number of well-known reportedly efficient scheduling algorithms . The results obtained demonstrate that the proposed algorithm is better than the other algorithms in terms of both solution quality and complexity . The major strength of the algorithm is the construction of the CPN-Dominate list in which the priorities of nodes are accurately captured . The list is used to generate an initial schedule upon which the local search process is invoked to refine and produce a final schedule . Nonetheless , there can be cases that the initial schedule is not good enough so that the local search process may get stuck in a poor local minimum point in the solution space . References -LSB- 1 -RSB- I. Ahmad , Y.K. Kwok and M.Y. Wu , ? Analysis , Evaluation , and Comparison of Algorithms for Scheduling Task Graphs on Parallel Processors , ? submitted for publication . FAST DSC MD ETF DLS 1.00 1.00 1.00 1.00 1.00 1.09 1.13 1.21 1.00 1.12 1.15 1.25 1.00 1.11 1.14 1.24 1.00 1.10 1.13 1.23 4 8 16 32 Algorithm -LRB- a -RRB- Normalized execution times of Laplace equation solver on the Intel Paragon . -LRB- b -RRB- Number of Processors used for the Laplace equation solver . FAST DSC MD ETF DLS 0.05 0.09 0.35 1.28 0.07 0.11 0.40 4.29 6.23 7.64 111.46 768.90 0.04 0.05 0.28 3.06 0.06 0.11 0.55 5.33 4 -LRB- 18 -RRB- 8 -LRB- 66 -RRB- 16 -LRB- 258 -RRB- 32 -LRB- 1026 -RRB- Algorithm Matrix Dimension -LRB- Number of Tasks -RRB- -LRB- c -RRB- Scheduling times -LRB- sec -RRB- on a SPARC Station 2 for the Laplace equation solver . Figure 6 : Normalized execution times , number of processors used , and scheduling algorithm running times for the Laplace equation solver for all the scheduling algorithms . Matrix Dimension FAST DSC MD ETF DLS 1 4 7 14 1 13 37 64 1 5 8 13 1 5 8 16 1 5 8 15 4 8 16 32 Algorithm Matrix Dimension FAST DSC MD ETF DLS 1.00 1.00 1.00 1.00 1.03 1.08 1.10 1.15 1.04 1.09 1.11 1.17 1.02 1.08 1.10 1.15 1.03 1.07 1.09 1.14 16 64 128 512 Algorithm -LRB- a -RRB- Normalized execution times of FFT on the Intel Paragon . -LRB- b -RRB- Number of Processors used for the FFT . FAST DSC MD ETF DLS 0.06 0.10 0.12 0.19 0.07 0.08 0.07 0.10 6.38 9.09 9.87 75.17 0.05 0.08 0.09 0.16 0.05 0.18 0.20 0.67 16 -LRB- 14 -RRB- 64 -LRB- 34 -RRB- 128 -LRB- 82 -RRB- 512 -LRB- 194 -RRB- Algorithm Number of Points -LRB- Number of Tasks -RRB- -LRB- c -RRB- Scheduling times -LRB- sec -RRB- on a SPARC Station 2 for FFT . Figure 7 : Normalized execution times , number of processors used , and scheduling algorithm running times for FFT for all the scheduling algorithms . Numer of Points FAST DSC MD ETF DLS 45 12 9 23 5 12 13 25 5 10 6 21 3 10 11 11 7 10 11 11 16 64 128 512 Algorithm Numer of Points FAST DSC ETF DLS 1.00 1.00 1.00 1.00 1.08 1.07 1.11 1.12 0.98 0.97 0.97 0.97 0.98 0.98 0.97 0.98 2000 3000 4000 5000 Algorithm -LRB- a -RRB- Normalized schedule lengths for random DAGs . -LRB- b -RRB- Number of Processors used for random DAGs . FAST DSC ETF DLS 42.01 46.05 72.36 97.79 40.37 48.36 86.26 121.98 748.71 699.83 966.80 2306.42 814.31 880.36 1182.06 2506.68 2000 3000 4000 5000 -LRB- 81049 -RRB- -LRB- 96847 -RRB- -LRB- 140821 -RRB- -LRB- 179902 -RRB- Algorithm Number of Nodes -LRB- Number of Edges -RRB- -LRB- c -RRB- Scheduling times -LRB- sec -RRB- on a SPARC Station 2 for random DAGs . Figure 8 : Normalized schedule lengths , number of processors used and scheduling times for the random DAGs for all the scheduling algorithms . Number of Nodes FAST DSC ETF DLS 141 195 174 219 1577 2253 3036 3758 123 139 162 198 131 197 166 212 2000 3000 4000 5000 Algorithm Number of Nodes -LSB- 2 -RSB- I. Ahmad , Y.K. Kwok , M.Y. Wu and W. Shu , ? CASCH : A Software Tool for Automatic Parallelization and Scheduling of Program on Multiprocessors , ? submitted for publication . -LSB- 3 -RSB- E.G. Coffman , Computer and Job-Shop Scheduling Theory , Wiley , New York , 1976 . -LSB- 4 -RSB- R.L. Graham , E.L. Lawler , J.K. Lenstra , and A.H.G. Rinnoy Kan , ? Optimization and Approximation in Deterministic Sequencing and Scheduling : A Survey , ? Annals of Discrete Mathematics , no. 5 , pp . 287-326 , 1979 . -LSB- 5 -RSB- J.Gu. , ? Parallel algorithms and architectures for very fast search , ? -LRB- PhD Thesis -RRB- Technical Report UUCS-TR-88-005 , Dept. of Computer Science , Univ. of Utah , 1988 . -LSB- 6 -RSB- J.Gu , ? Local Search for the Satisfiability -LRB- SAT -RRB- Problem ? , IEEE Transactions on Systems and Cybernetics , vol . 23 , no. 3 , Jul. / Aug. . 1993 , pp. 1108-1129 . -LSB- 7 -RSB- T.C. Hu , ? Parallel Sequencing and Assembly Line Problems , ? Operations Research , vol . 19 , no. 6 , pp. 841 - 848 , Nov. 1961 . -LSB- 8 -RSB- J.J.Hwang , Y.C. Chow , F.D. Anger and C.Y. Lee , ? Scheduling Precedence Graphs in Systems with Interprocessor Communication Times , ? SIAM Journal of Computing , vol . 18 , no. 2 , pp. 244-257 , Apr. 1989 . -LSB- 9 -RSB- H.Kasahara and S. Narita , ? Practical Multiprocessor Scheduling Algorithms for Efficient Parallel Processing , ? IEEE Trans . on Computers , vol . C-33 , pp. 1023-1029 , Nov. 1984 . -LSB- 10 -RSB- R.E.Lord , J.S. Kowalik and S.P. Kumar , ? Solving Linear Algebraic Equations on an MIMD Computer , ? Journal of the ACM , 30 -LRB- 1 -RRB- , pp . 103-117 , Jan. 1983 . -LSB- 11 -RSB- S.S. Pande , D.P. Agrawal and J. Mauney , ? A Threshold Scheduling Strategy for Sisal on Distributed Memory Machines , ? Journal of Parallel and Distributed Computing , 21 , pp. 223-236 , 1994 . -LSB- 12 -RSB- C.H.Papadimitriou and K. Steiglitz , Combinatorial Optimization : Algorithms and Complexity , Prentice-Hall , Englewood Cliffs , NJ , 1982 . -LSB- 13 -RSB- B.Shirazi , M. Wang and G. Pathak , ? Analysis and Evaluation of Heuristic Methods for Static Scheduling , ? Journal of Parallel and Distributed Computing , no. 10 , pp. 222-232 , 1990 . -LSB- 14 -RSB- G.C.Sih and E.A. Lee , ? A Compile-Time Scheduling Heuristic for Interconnection-Constrained Heterogeneous Processor Architectures , ? IEEE Trans . on Parallel and Distributed Systems , vol . 4 , no. 2 , pp . 75-87 , Feb. 1993 . -LSB- 15 -RSB- R. Sosic and J. Gu , ? Fast search algorithms for the n-queens problem , ? IEEE Trans . on Systems , Man , and Cybernetics , SMC-21 -LRB- 6 -RRB- :1572 -1576 , Nov. / Dec. 1991 . -LSB- 16 -RSB- R.Sosic and J.Gu , ? Efficient Local Search with Conflict Minimizations , ? IEEE Transactions on Data and Knowledge Engineering , Vol . 6 , No. 4 , pp. 661-668 , Aug. 1994 . -LSB- 17 -RSB- M.Y. Wu and D.D. Gajski , ? Hypertool : A Programming Aid for Message-Passing Systems , ? IEEE Trans . on Parallel and Distributed Systems , vol . 1 , no. 3 , pp. 330-343 , Jul. 1990 . -LSB- 18 -RSB- T. Yang and A. Gerasoulis , ? DSC : Scheduling Parallel Tasks on an Unbounded Number of Processors , ? IEEE Trans . on Parallel and Distributed Systems , vol . 5 , no. 9 , pp. 951-967 , Sep. 1994 .

O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O B I O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

local search
multiprocessors
parallel processing
scheduling
task graphs