A New Deterministic Parallel Sorting Algorithm With an Experimental Evaluation David R. Helman Joseph J?aJ?a ? David A. Badery Institute for Advanced Computer Studies & Department of Electrical Engineering , University of Maryland , College Park , MD 20742 fhelman , joseph , dbaderg@umiacs.umd.edu August 15 , 1996 Abstract We introduce a new deterministic parallel sorting algorithm based on the regular sampling approach . The algorithm uses only two rounds of regular all-to-all personalized communication in a scheme that yields very good load balancing with virtually no overhead . Moreover , unlike previous variations , our algorithm efficiently handles the presence of duplicate values without the overhead of tagging each element with a unique identifier . This algorithm was implemented in Split-C and run on a variety of platforms , including the Thinking Machines CM-5 , the IBM SP-2-WN , and the Cray Research T3D . We ran our code using widely different benchmarks to examine the dependence of our algorithm on the input distribution . Our experimental results illustrate the efficiency and scalability of our algorithm across different platforms . In fact , the performance compares closely to that of our random sample sort algorithm , which seems to outperform all similar algorithms known to the authors on these platforms . Together , their performance is nearly invariant over the set of input distributions , unlike previous efficient algorithms . However , unlike our randomized sorting algorithm , the performance and memory requirements of our regular sorting algorithm can be deterministically guaranteed . ? Supported in part by NSF grant No . CCR-9103135 and NSF HPCC/GCAG grant No . BIR-9318183 . yThe support by NASA Graduate Student Researcher Fellowship No . NGT-50951 is gratefully acknowledged . 1 Introduction We present a novel variation on the approach of sorting by regular sampling which leads to a new deterministic sorting algorithm that achieves optimal computational speedup with very little communication -LSB- 15 -RSB- . Our algorithm exchanges the single step of irregular communication used by previous implementations for two steps of regular communication . In return , our algorithm reduces the problem of poor load balancing because it is able to sustain a high sampling rate at substantially less cost . In addition , our algorithm efficiently accommodates the presence of duplicates without the overhead of tagging each element . And our algorithm achieves predictable , regular communication requirements which are essentially invariant with respect to the input distribution . Utilizing regular communication has become more important with the advent of message passing standards , such as MPI -LSB- 17 -RSB- , which seek to guarantee the availability of very efficient -LRB- often machine specific -RRB- implementations of certain basic collective communication routines . Our algorithm was implemented in a high-level language and run on a variety of platforms , including the Thinking Machines CM-5 , the IBM SP-2 , and the Cray Research T3D . We ran our code using a variety of benchmarks that we identified to examine the dependence of our algorithm on the input distribution . Our experimental results are consistent with the theoretical analysis and illustrate the efficiency and scalability of our algorithm across different platforms . In fact , the performance compares closely to that of our random sample sort algorithm , which seems to outperform all similar algorithms known to the authors on these platforms . Together , their performance is nearly indifferent to the set of input distributions , unlike previous efficient algorithms . However , unlike our randomized sorting algorithm , the performance and memory requirements of our regular sorting algorithm can be guaranteed with probability one . The high-level language used in our studies is Split-C -LSB- 10 -RSB- , an extension of C for distributed memory machines . The algorithm makes use of MPI-like communication primitives but does not make any assumptions as to how these primitives are actually implemented . The basic data transport is a read or write operation . The remote read and write typically have both blocking and nonblocking versions . Also , when reading or writing more than a single element , bulk data transports are provided with corresponding bulk read and bulk write primitives . Our collective communication primitives , described in detail in -LSB- 6 -RSB- , are similar to those of the MPI -LSB- 17 -RSB- , the IBM POWERparallel -LSB- 7 -RSB- , and the Cray MPP systems -LSB- 9 -RSB- and , for example , include the following : transpose , bcast , gather , and scatter . Brief descriptions of these are as follows . The transpose primitive is an all-to-all personalized communication in which each processor has to send a unique block of data to every processor , and all the blocks are of the same size . The bcast primitive is used to copy a block of data from a single source to all the other processors . The primitives gather and scatter are companion primitives . Scatter divides a single array residing on a processor into equal-sized blocks , each of which is distributed to a unique processor , and gather coalesces these blocks back into a single array at a particular processor . See -LSB- 3 , 6 , 4 , 5 -RSB- for algorithmic details , performance analyses , and empirical results for these communication primitives . The organization of this paper is as follows . Section 2 presents our computation model for analyzing parallel algorithms . Section 3 describes in detail our improved sample sort algorithm . Finally , Section 4 describes our data sets and the experimental performance of our sorting algorithm . 2 The Parallel Computation Model We use a simple model to analyze the performance of our parallel algorithms . Our model is based on the fact that current hardware platforms can be viewed as a collection of powerful processors connected by a communication network that can be modeled as a complete graph on which communication is subject to the restrictions imposed by the latency and the bandwidth properties of the network . We view a parallel algorithm as a sequence of local computations interleaved with communication steps , where we allow computation and communication to overlap . We account for communication costs as follows . Assuming no congestion , the transfer of a block consisting of m contiguous words between two processors takes -LRB- o + oem -RRB- time , where o is the latency of the network and oe is the time per word at which a processor can inject or receive data from the network . Note that the bandwidth per processor is inversely proportional to oe . We assume that the bisection bandwidth is sufficiently high to support block permutation routing amongst the p processors at the rate of 1oe . In particular , for any subset of q processors , a block permutation amongst the q processors takes -LRB- o + oem -RRB- time , where m is the size of the largest block . Using this cost model , we can evaluate the communication time Tcomm -LRB- n ; p -RRB- of an algorithm as a function of the input size n , the number of processors p , and the parameters o and oe . The coefficient of o gives the total number of times collective communication primitives are used , and the coefficient of oe gives the maximum total amount of data exchanged between a processor and the remaining processors . This communication model is close to a number of similar models -LRB- e.g. -LSB- 11 , 19 , 1 -RSB- -RRB- that have recently appeared in the literature and seems to be well-suited for designing parallel algorithms on current high performance platforms . We define the computation time Tcomp as the maximum time it takes a processor to perform all the local computation steps . In general , the overall performance Tcomp + Tcomm involves a tradeoff between Tcomp and Tcomm . In many cases , it is possible to minimize both Tcomp and Tcomm simultaneously , and sorting is such a case . 3 A New Sorting Algorithm by Regular Sampling Consider the problem of sorting n elements equally distributed amongst p processors , where we assume without loss of generality that p divides n evenly . The idea behind sorting by regular sampling is to find a set of p ? 1 splitters to partition the n input elements into p groups indexed from 1 up to p such that every element in the ith group is less than or equal to each of the elements in the -LRB- i + 1 -RRB- th group , for -LRB- 1 < = i < = p ? 1 -RRB- . Then the task of sorting each of the p groups can be turned over to the correspondingly indexed processor , after which the n elements will be arranged in sorted order . The efficiency of this algorithm obviously depends on how well we divide the input , and this in turn depends on how evenly we choose the splitters . One way to choose the splitters is by regularly sampling the sorted input elements at each processor - hence the name Sorting by Regular Sampling . A previous version of regular sample sort -LSB- 18 , 16 -RSB- , known as Parallel Sorting by Regular Sampling -LRB- PSRS -RRB- , first sorts the np elements at each processor and then selects every ? np2 ? th element as a sample . These samples are then routed to a single processor , where they are sorted and every pth sample is selected as a splitter . Each processor then uses these splitters to partition the sorted input values and then routes the resulting subsequences to the appropriate destinations , after which local merging of these subsequences is done to complete the sorting process . The first difficulty with this approach is the load balance . There exist inputs for which at least one processor will be left with as many as ? 2np ? np2 ? p + 1 ? elements at the completion of sorting . This could be reduced by choosing more samples , but this would also increase the overhead . And no matter how many samples are chosen , previous studies have shown that the load balance would still deteriorate linearly with the number of duplicates -LSB- 16 -RSB- . One could , of course , tag each item with a unique value , but this would also double the cost of both memory access and interprocessor communication . The other difficulty is that no matter how the routing is scheduled , there exist inputs that give rise to large variations in the number of elements destined for different processors , and this in turn results in an inefficient use of the communication bandwidth . Moreover , such an irregular communication scheme can not take advantage of the regular communication primitives proposed under the MPI standard -LSB- 17 -RSB- . In our algorithm , which is parameterized by a sampling ratio s ? p < = s < = np2 ? , we guarantee that , at the completion of sorting , each processor will have at most ? np + ns ? p ? elements , while incurring no overhead in gathering the set of samples used to identify the splitters . This bound holds regardless of the number of duplicate elements present in the input . Moreover , we are able to replace the irregular routing with exactly two calls to our transpose primitive . The pseudocode for our algorithm is as follows : ffl Step -LRB- 1 -RRB- : Each processor Pi -LRB- 1 < = i < = p -RRB- sorts each of its np input values using an appropriate sequential sorting algorithm . For integers we use the radix sort algorithm , whereas for floating point numbers we use the merge sort algorithm . The sorted data is then \ dealt '' into p bins so that the kth item in the sorted order is placed into the ? jk ? 1p k + 1 ? th position of the -LRB- -LRB- -LRB- k ? 1 -RRB- mod p -RRB- + 1 -RRB- th bin . ffl Step -LRB- 2 -RRB- : Each processor Pi routes the contents of bin j to processor Pj , for -LRB- 1 < = i ; j < = p -RRB- , which is equivalent to performing a transpose operation with block size np2 . ffl Step -LRB- 3 -RRB- : From each of the p sorted subsequences received in Step -LRB- 2 -RRB- , processor Pp selects each ? k np2 s ? th element as a sample , for -LRB- 1 < = k < = s -RRB- and a given value of s ? p < = s < = np2 ? . ffl Step -LRB- 4 -RRB- : Processor Pp merges the p sorted subsequences of samples and then selects each -LRB- ks -RRB- th sample as Splitter -LSB- k -RSB- , for -LRB- 1 < = k < = p ? 1 -RRB- . By default , the pth splitter is the largest value allowed by the data type used . Additionally , binary search is used to compute for the set of samples Sk with indices -LRB- ks ? s + 1 -RRB- through -LRB- ks -RRB- the number of samples Est -LSB- k -RSB- which share the same value as Splitter -LSB- k -RSB- . ffl Step -LRB- 5 -RRB- : Processor Pp broadcasts the Splitter and Est arrays to the other p ? 1 processors . ffl Step -LRB- 6 -RRB- : Each processor Pi uses binary search to define for each of the p sorted sequences received in Step -LRB- 2 -RRB- and each of the p splitters a subsequence T -LRB- j ; k -RRB- . The set of p subsequences fT -LRB- j ; 1 -RRB- ; T -LRB- j ; 2 -RRB- ; : : : ; T -LRB- j ;p -RRB- g associated with Splitter -LSB- j -RSB- all contain values which are greater than or equal to Splitter -LSB- j ? 1 -RSB- and less then or equal to Splitter -LSB- j -RSB- , and collectively include at most ? Est -LSB- j -RSB- ? np2 s ? elements with the same value as Splitter -LSB- j -RSB- . ffl Step -LRB- 7 -RRB- : Each processor Pi routes the p subsequences associated with Splitter -LSB- j -RSB- to processor Pj , for -LRB- 1 < = i ; j < = p -RRB- . Since no two processors will exchange more than ? np2 + nsp ? elements , this is equivalent to performing a transpose operation with block size ? np2 + nsp ? . ffl Step -LRB- 8 -RRB- : Each processor Pi \ unshuffles '' all those subsequences sharing a common origin in Step -LRB- 2 -RRB- . ffl Step -LRB- 9 -RRB- : Each processor Pi merges the p consolidated subsequences to produce the ith column of the sorted array . Before establishing the complexity of this algorithm , we need to establish the following theorem . Theorem 1 : The number of elements sent by processor Pi to processor Pj in Step -LRB- 7 -RRB- is at most ? np2 + nsp ? p ? for i = p and ? np2 + nsp ? for i < p. Consequently , at the completion of the algorithm , no processor holds more than ? np + ns ? p ? elements , for n > = p3 and ? p < = s < = np2 ? . Proof : Let Sj be the set of samples from the sorted array of samples in Step -LRB- 4 -RRB- with indices -LRB- js?s +1 -RRB- through -LRB- js -RRB- , inclusively . Let S -LRB- j ; i -RRB- be the subset of samples in Sj originating from processor Pi , and let c -LRB- j ; i -RRB- be the cardinality of S -LRB- j ; i -RRB- . Let Vj = Splitter -LSB- j -RSB- , let c -LRB- j ; i ; 1 -RRB- be the number of samples in S -LRB- j ; i -RRB- with value less than Vj , and let c -LRB- j ; i ; 2 -RRB- = -LRB- c -LRB- j ; i -RRB- ? c -LRB- j ; i ; 1 -RRB- -RRB- be the number of samples in S -LRB- j ; i -RRB- with value Vj . Let R -LRB- j ; i -RRB- = f -LSB- S -LRB- t ; i -RRB- : 1 < = t < -LRB- j ? 1 -RRB- g , let b -LRB- j ; i -RRB- = Pj ? 1 t = 0 c -LRB- t ; i -RRB- be the cardinality of R -LRB- j ; i -RRB- , let b -LRB- j ; i ; 1 -RRB- be the number of samples in R -LRB- j ; i -RRB- with value less than Vj , and let b -LRB- j ; i ; 2 -RRB- = -LRB- b -LRB- j ; i -RRB- ? b -LRB- j ; i ; 1 -RRB- -RRB- be the number of samples in R -LRB- j ; i -RRB- with value Vj . Obviously , b -LRB- j ; i ; 2 -RRB- will only be nonzero if Splitter -LSB- j ? 1 -RSB- = Vj . Finally , for simplicity of discussion but without loss of generality , we assume that n is a constant multiple of p2s . Clearly , each sample can be mapped in a one-to-one fashion to the sorted input generated during Step -LRB- 1 -RRB- -LRB- but before being distributed amongst the bins -RRB- . For example , the first sample in S -LRB- j ; i -RRB- maps to the ?? b -LRB- j ; i -RRB- + 1 ? nps ? th element in the sorted input at processor Pi , the second sample in S -LRB- j ; i -RRB- maps to the ?? b -LRB- j ; i -RRB- + 2 ? nps ? th element in the sorted input at processor Pi , and so forth up to the cth -LRB- j ; i -RRB- element which maps to the ?? b -LRB- j ; i -RRB- + c -LRB- j ; i -RRB- ? nps ? th element in the sorted input at processor Pi . Hence , it follows that L -LRB- j ; i -RRB- elements in the sorted input of Step -LRB- 1 -RRB- at processor Pi will be less than Vj , where ?? b -LRB- j ; i ; 1 -RRB- + c -LRB- j ; i ; 1 -RRB- ? nps < = L -LRB- j ; i -RRB- < = ?? b -LRB- j ; i ; 1 -RRB- + c -LRB- j ; i ; 1 -RRB- + 1 ? nps ? 1 ?? . It is also true that at least M -LRB- j ; i -RRB- elements in the sorted input of Step -LRB- 1 -RRB- will be less than or equal to Vj , where M -LRB- j ; i -RRB- = ? b -LRB- j ; i -RRB- + c -LRB- j ; i -RRB- ? nps . The shuffling of Step -LRB- 1 -RRB- together with the transpose of Step -LRB- 2 -RRB- maps the tth element at processor Pi into the ? j t ? 1p k + 1 ? th position of the ith subarray at processor P -LRB- -LRB- -LRB- t ? 1 -RRB- mod p -RRB- +1 -RRB- , a subarray which we will denote as SA -LRB- -LRB- -LRB- -LRB- t ? 1 -RRB- mod p -RRB- +1 -RRB- ; i -RRB- . Now , L -LRB- j ; r ; i -RRB- elements in SA -LRB- r ; i -RRB- will be less than Vj and will unequivocally route to processors P1 through Pj , where : ? b -LRB- j ; i ; 1 -RRB- + c -LRB- j ; i ; 1 -RRB- ? n p2s < = L -LRB- j ; r ; i -RRB- < = ? b -LRB- j ; i ; 1 -RRB- + c -LRB- j ; i ; 1 -RRB- + 1 ? n p2s if r < p ? b -LRB- j ; i ; 1 -RRB- + c -LRB- j ; i ; 1 -RRB- ? n p2s < = L -LRB- j ; r ; i -RRB- < = ?? b -LRB- j ; i ; 1 -RRB- + c -LRB- j ; i ; 1 -RRB- + 1 ? n p2s ? 1 ? if r = p Furthermore , at least M -LRB- j ; r ; i -RRB- elements in SA -LRB- r ; i -RRB- will be less than or equal to Vj , where M -LRB- j ; r ; i -RRB- = -LRB- b -LRB- j ; i -RRB- + c -LRB- j ; i -RRB- -RRB- np2 s -RRB- . This means that the p subarrays at processor Pr collectively have at least pX t = 1 M -LRB- j ; r ; t -RRB- = pX t = 1 ? b -LRB- j ; t -RRB- + c -LRB- j ; t -RRB- ? n p2s = j n p2 elements which are greater than or equal to Vj . Furthermore , pX t = 1 ? M -LRB- j ; r ; t -RRB- ? L -LRB- j ; r ; t -RRB- ? < = pX t = 1 ?? b -LRB- j ; t -RRB- + c -LRB- j ; t -RRB- ? n p2s ? ? b -LRB- j ; t ; 1 -RRB- + c -LRB- j ; t ; 1 -RRB- ? n p2s ? = pX t = 1 ? b -LRB- j ; t ; 2 -RRB- + c -LRB- j ; t ; 2 -RRB- ? n p2s = -LRB- Est ? -LSB- j -RSB- + Est -LSB- j -RSB- -RRB- ? n p2s ; where ? Est -LSB- j -RSB- ? np2 s ? is the number of elements equal to Vj which the algorithm in Step -LRB- 6 -RRB- will seek to route to processor Pj and ? Est ? -LSB- j -RSB- ? np2 s ? is the number of elements equal to Vj which the algorithm in Step -LRB- 6 -RRB- will seek to route to processors P1 through P -LRB- j ? 1 -RRB- . From this it follows that the algorithm will always be able to route a minimum of Min -LRB- j ; r -RRB- = Ppt = 1 M -LRB- j ; r ; t -RRB- = j np2 elements to processors P1 through Pj . On the other hand , the maximum number of elements that will be routed by this algorithm to these processors is : Max -LRB- j ; r -RRB- = pX t = 1 L -LRB- j ; r ; t -RRB- ! + -LRB- Est ? -LSB- j -RSB- + Est -LSB- j -RSB- -RRB- ? n p2s < = 8 >> >> > < >> >> > : Ppt = 1 ?? b -LRB- j ; t ; 1 -RRB- + c -LRB- j ; t ; 1 -RRB- + 1 ? np2 s + ? b -LRB- j ; t ; 2 -RRB- + c -LRB- j ; t ; 2 -RRB- ? np2 s ? = -LRB- js + p -RRB- np2 s if r < p Ppt = 1 ??? b -LRB- j ; t ; 1 -RRB- + c -LRB- j ; t ; 1 -RRB- + 1 ? np2 s ? 1 ? + ? b -LRB- j ; t ; 2 -RRB- + c -LRB- j ; t ; 2 -RRB- ? np2 s ? = ? -LRB- js + p -RRB- np2 s ? p ? if r = p Hence , the maximum number of elements send by processor Pr to processor Pj is : Max -LRB- j ; r -RRB- ? Min -LRB- j ? 1 ; r -RRB- = 8 >> >> > < >> >> > : -LRB- js + p -RRB- np2 s ? -LRB- -LRB- j ? 1 -RRB- s -RRB- np2 s = ? np2 + nps ? if r < p ? -LRB- js + p -RRB- np2 s ? p ? ? -LRB- -LRB- j ? 1 -RRB- s -RRB- np2 s = ? np2 + nps ? p ? if r = p and Theorem 1 follows . With the results of Theorem 1 , the analysis of our algorithm for sorting by regular sampling is as follows . Steps -LRB- 3 -RRB- , -LRB- 4 -RRB- , -LRB- 6 -RRB- , -LRB- 8 -RRB- , and -LRB- 9 -RRB- require O -LRB- sp -RRB- , O -LRB- sp log p -RRB- , O ? p2 log p ? , O ? np + ns + p2 ? p ? , and O ?? np + ns ? p ? log p ? time , respectively . The cost of sequential sorting in Step -LRB- 1 -RRB- depends on the data type - sorting integers using radix sort requires O ? np ? time , whereas sorting floating point numbers using merge sort requires O ? np log ? np ?? time . Steps -LRB- 2 -RRB- , -LRB- 5 -RRB- , and -LRB- 7 -RRB- call the communication primitives transpose , bcast , and transpose , respectively . The analysis of these primitives in -LSB- 6 -RSB- shows that these three steps require Tcomm -LRB- n ; p -RRB- < = ? o + np2 -LRB- p ? 1 -RRB- oe ? , Tcomm -LRB- n ; p -RRB- < = -LRB- o +2 -LRB- p ? 1 -RRB- oe -RRB- , and Tcomm -LRB- n ; p -RRB- < = ? o + ? np2 + nsp ? -LRB- p ? 1 -RRB- oe ? , respectively . Hence , with high probability , the overall complexity of our sorting algorithm is given -LRB- for floating point numbers -RRB- by T -LRB- n ; p -RRB- = Tcomp -LRB- n ; p -RRB- + Tcomm -LRB- n ; p -RRB- = O ? n p log n + o + n p oe ? -LRB- 1 -RRB- for n > = p3 and ? p < = s < = np2 ? . Clearly , our algorithm is asymptotically optimal with very small coefficients . But a theoretical comparison of our running time with previous sorting algorithms is difficult , since there is no consensus on how to model the cost of the irregular communication used by the most efficient algorithms . Hence , it is very important to perform an empirical evaluation of an algorithm using a wide variety of benchmarks , as we will do next . 4 Performance Evaluation Our sample sort algorithm was implemented using Split-C -LSB- 10 -RSB- and run on a variety of machines and processors , including the Cray Research T3D , the IBM SP-2-WN , and the Thinking Machines CM-5 . For every platform , we tested our code on nine different benchmarks , each of which had both a 32-bit integer version -LRB- 64-bit on the Cray T3D -RRB- and a 64-bit double precision floating point number -LRB- double -RRB- version . 4.1 Sorting Benchmarks Our nine sorting benchmarks are defined as follows , in which n and p are assumed for simplicity but without loss of generality to be powers of two and MAXD , the maximum value allowed for doubles , is approximately 1:8 ? 10308 . 1 . Uniform -LSB- U -RSB- , a uniformly distributed random input , obtained by calling the C library random number generator random -LRB- -RRB- . This function , which returns integers in the range to ? 231 ? 1 ? , is seeded by each processor Pi with the value -LRB- 21 +1001 i -RRB- . For the double data type , we \ normalize '' the integer benchmark values by first subtracting the value 230 and then scaling the result by ? 2 ? 30 ? MAXD ? . 2 . Gaussian -LSB- G -RSB- , a Gaussian distributed random input , approximated by adding four calls to random -LRB- -RRB- and then dividing the result by four . For the double data type , we normalize the integer benchmark values in the manner described for -LSB- U -RSB- . 3 . Zero -LSB- Z -RSB- , a zero entropy input , created by setting every value to a constant such as zero . 4 . Bucket Sorted -LSB- B -RSB- , an input that is sorted into p buckets , obtained by setting the first np2 elements at each processor to be random numbers between and -LRB- 231 p ? 1 -RRB- , the second np2 elements at each processor to be random numbers between 231 p and -LRB- 232 p ? 1 -RRB- , and so forth . For the double data type , we normalize the integer benchmark values in the manner described for -LSB- U -RSB- . 5 . g-Group -LSB- g-G -RSB- , an input created by first dividing the processors into groups of consecutive processors of size g , where g can be any integer which partitions p evenly . If we index these groups in consecutive order from 1 up to pg , then for group j we set the first npg elements to be random numbers between ??? -LRB- j ? 1 -RRB- g + p2 ? 1 ? mod p ? + 1 ? 231 p and ???? -LRB- j ? 1 -RRB- g + p2 ? mod p ? + 1 ? 231 p ? 1 ? , the second npg elements at each processor to be random numbers between ??? -LRB- j ? 1 -RRB- g + p2 ? mod p ? + 1 ? 231 p and ???? -LRB- j ? 1 -RRB- g + p2 + 1 ? mod p ? + 1 ? 231 p ? 1 ? , and so forth . For the double data type , we normalize the integer benchmark values in the manner described for -LSB- U -RSB- . 6 . Staggered -LSB- S -RSB- , created as follows : if the processor index i is less than or equal to p2 , then we set all np elements at that processor to be random numbers between ? -LRB- 2i ? 1 -RRB- 231 p ? and ? -LRB- 2i -RRB- 231 p ? 1 ? . Otherwise , we set all np elements to be random numbers between ? -LRB- 2i ? p ? 2 -RRB- 231 p ? and ? -LRB- 2i ? p ? 1 -RRB- 231 p ? 1 ? . For the double data type , we normalize the integer benchmark values in the manner described for -LSB- U -RSB- . 7 . Worst-Load Regular -LSB- WR -RSB- - an input consisting of values between and -LRB- 231 ? 1 -RRB- designed to induce the worst possible load balance at the completion of our regular sorting . Specifically , at the completion of sorting , the odd-indexed processors will hold -LRB- np + ns ? p -RRB- elements , whereas the evenindexed processors will hold -LRB- np ? ns + p -RRB- elements . The benchmark is defined as follows . At processor P1 , for odd values of j between 1 and -LRB- p ? 2 -RRB- , the elements with indices ? -LRB- j ? 1 -RRB- np2 + 1 ? through ? j np2 ? 1 ? are set to random values between ? -LRB- j ? 1 -RRB- 231 p + 1 ? and ? j 231 p ? 1 ? , the elements with indices ? j np2 ? through ? j np2 + nsp ? 1 ? are set to ? j 231 p ? , the elements with indices ? j np2 + nsp ? through ? -LRB- j + 1 -RRB- np2 ? 1 ? are set to random values between ? j 231 p + 1 ? and ? -LRB- j + 1 -RRB- 231 p ? 1 ? , and the element with index ? -LRB- j + 1 -RRB- 231 p ? is set to ? -LRB- j + 1 -RRB- 231 p ? . At processor P1 , for j equal to -LRB- p ? 1 -RRB- , the elements with indices ? -LRB- j ? 1 -RRB- np2 + 1 ? through ? j np2 ? 1 ? are set to random values between ? -LRB- j ? 1 -RRB- 231 p + 1 ? and ? j 231 p ? 1 ? , the elements with indices ? j np2 ? through ? j np2 + nsp ? 1 ? are set to ? j 231 p ? , and the elements with indices ? j np2 + nsp ? through ? -LRB- j + 1 -RRB- np2 ? are set to random values between ? j 231 p + 1 ? and ? -LRB- j + 1 -RRB- 231 p ? 1 ? . At processor Pi -LRB- i > 1 -RRB- , for odd values of j between 1 and p , the elements with indices ? -LRB- j ? 1 -RRB- np2 + 1 ? through ? j np2 + nsp ? 1 ? are set to random values between ? -LRB- j ? 1 -RRB- 231 p + 1 ? and ? j 231 p ? 1 ? , the elements with index ? j np2 + nsp ? is set to ? j 231 p + i ? , and the elements with indices ? j np2 + nsp + 1 ? through ? -LRB- j + 1 -RRB- np2 ? are set to random values between ? j 231 p + 1 + i ? and ? -LRB- j + 1 -RRB- 231 p ? 1 ? . For the double data type , we normalize the integer benchmark values in the manner described for -LSB- U -RSB- . 8 . Deterministic Duplicates -LSB- DD -RSB- , an input of duplicates in which we set all np elements at each of the first p2 processors to be log n , all np elements at each of the next p4 processors to be log ? n2 ? , and so forth . At processor Pp , we set the first n2p elements to be log ? np ? , the next n4p elements to be log ? n2p ? , and so forth . 9 . Randomized Duplicates -LSB- RD -RSB- , an input of duplicates in which each processor fills an array T with some constant number range -LRB- range is 32 for our work -RRB- of random values between and -LRB- range ? 1 -RRB- whose sum is S . The first T -LSB- 1 -RSB- S ? np values of the input are then set to a random value between and -LRB- range ? 1 -RRB- , the next T -LSB- 2 -RSB- S ? np values of the input are then set to another random value between and -LRB- range ? 1 -RRB- , and so forth . See -LSB- 14 -RSB- for a detailed justification of these benchmarks . 4.2 Experimental Results For each experiment , the input is evenly distributed amongst the processors . The output consists of the elements in non-descending order arranged amongst the processors so that the elements at each processor are in sorted order and no element at processor Pi is greater than any element at processor Pj , for all i < j. Two variations were allowed in our experiments . First , radix sort was used to sequentially sort integers , whereas merge sort was used to sort double precision floating point numbers -LRB- doubles -RRB- . Second , different implementations of the communication primitives were allowed for each machine . Wherever possible , we tried to use the vendor supplied implementations . In fact , IBM does provide all of our communication primitives as part of its machine specific Collective Communication Library -LRB- CCL -RRB- -LSB- 7 -RSB- and MPI . As one might expect , they were faster than the high level Split-C implementation . Optimal Number of Samples s for Sorting on T3D Number of Processors int . / proc . 8 16 32 64 128 16K 128 128 128 128 128 32K 128 128 128 128 128 64K 256 256 256 256 128 128K 256 256 256 256 256 256K 512 512 512 256 512 512K 512 512 512 512 512 1M 1024 512 512 512 1024 Table I : Optimal number of samples s for sorting the -LSB- WR -RSB- integer benchmark on the Cray T3D , for a variety of processors and input sizes . Optimal Number of Samples s for Sorting on SP2 Number of Processors int . / proc . 8 16 32 64 128 16K 256 128 128 128 128 32K 256 256 256 256 256 64K 512 256 256 256 512 128K 512 512 512 512 512 256K 512 512 512 256 512 512K 1024 1024 1024 1024 1024 1M 1024 1024 1024 1024 1024 Table II : Optimal number of samples s for sorting the -LSB- WR -RSB- integer benchmark on the IBM SP-2-WN , for a variety of processors and input sizes . Tables I and II examine the preliminary question of the optimal number of samples s for sorting on the Cray T3D and the IBM SP-2-WN . They show the value of s which achieved the best performance on the Worst-Load Regular -LSB- WR -RSB- benchmark , as a function of both the number of processors p and the number of keys per processor np . The results suggest that a good rule for choosing s is to set it to 2b 12 log -LRB- n =p -RRB- c ss qnp , which is what we do for the remainder of this discussion . To compare this choice for s with the theoretical expectation , we recall that the complexity of Step -LRB- 3 -RRB- is O -LRB- sp log p -RRB- , whereas the complexity of Step -LRB- 9 -RRB- is O ?? np + ns ? p ? log p ? . Hence , the first term is an increasing function of s , whereas the second term is a decreasing function of s . It is easy to verify that the expression for the sum of these two complexities is minimized for s = O ? qnp ? , and , hence , the theoretical expectation for the optimal value of s agrees with what we observe experimentally . Size -LSB- U -RSB- -LSB- G -RSB- -LSB- 2-G -RSB- -LSB- 4-G -RSB- -LSB- B -RSB- -LSB- S -RSB- -LSB- Z -RSB- -LSB- WR -RSB- -LSB- DD -RSB- -LSB- RD -RSB- 256K 0.047 0.046 0.040 0.040 0.046 0.042 0.036 0.051 0.037 0.042 1M 0.104 0.102 0.094 0.092 0.103 0.094 0.080 0.113 0.081 0.089 4M 0.309 0.305 0.299 0.291 0.310 0.303 0.245 0.325 0.250 0.261 16M 1.09 1.08 1.09 1.06 1.10 1.11 0.903 1.13 0.904 0.930 64M 4.18 4.11 4.22 4.09 4.15 4.31 3.52 4.21 3.52 3.59 Table III : Total execution time -LRB- in seconds -RRB- required to sort a variety of integer benchmarks on a 64-node Cray T3D . Size -LSB- U -RSB- -LSB- G -RSB- -LSB- 2-G -RSB- -LSB- 4-G -RSB- -LSB- B -RSB- -LSB- S -RSB- -LSB- Z -RSB- -LSB- WR -RSB- -LSB- DD -RSB- -LSB- RD -RSB- 256K 0.055 0.055 0.050 0.048 0.051 0.049 0.046 0.056 0.047 0.050 1M 0.091 0.094 0.085 0.086 0.089 0.087 0.083 0.099 0.087 0.089 4M 0.237 0.236 0.229 0.223 0.224 0.228 0.222 0.253 0.231 0.239 16M 0.873 0.878 0.974 0.886 0.868 0.969 0.819 0.904 0.835 0.851 64M 3.45 3.46 3.83 3.86 3.38 3.79 3.09 3.45 3.11 3.12 Table IV : Total execution time -LRB- in seconds -RRB- required to sort a variety of integer benchmarks on a 64-node IBM SP-2-WN . Size -LSB- U -RSB- -LSB- G -RSB- -LSB- 2-G -RSB- -LSB- 4-G -RSB- -LSB- B -RSB- -LSB- S -RSB- -LSB- Z -RSB- -LSB- WR -RSB- -LSB- DD -RSB- -LSB- RD -RSB- 256K 0.056 0.056 0.046 0.046 0.055 0.045 0.044 0.060 0.043 0.050 1M 0.126 0.126 0.113 0.113 0.131 0.111 0.107 0.136 0.018 0.115 4M 0.411 0.411 0.387 0.394 0.416 0.389 0.376 0.435 0.383 0.384 16M 1.60 1.59 1.55 1.55 1.58 1.55 1.49 1.60 1.50 1.49 64M 6.53 6.57 6.44 6.45 6.55 6.49 6.26 6.61 6.26 6.14 Table V : Total execution time -LRB- in seconds -RRB- required to sort a variety of double benchmarks on a 64-node Cray T3D . Tables III , IV , V , and VI display the performance of our sample sort as a function of input distribution for a variety of input sizes . In each case , the performance is essentially independent of the input distribution . These figures present results obtained on a 64 node Cray T3D and a 64 node IBM Size -LSB- U -RSB- -LSB- G -RSB- -LSB- 2-G -RSB- -LSB- 4-G -RSB- -LSB- B -RSB- -LSB- S -RSB- -LSB- Z -RSB- -LSB- WR -RSB- -LSB- DD -RSB- -LSB- RD -RSB- 256K 0.090 0.087 0.082 0.080 0.084 0.080 0.077 0.093 0.081 0.084 1M 0.181 0.184 0.176 0.186 0.176 0.176 0.168 0.198 0.187 0.188 4M 0.598 0.590 0.580 0.576 0.578 0.600 0.570 0.614 0.584 0.589 16M 2.26 2.25 2.35 2.35 2.26 2.40 2.25 2.34 2.29 2.33 64M 9.61 9.61 10.0 10.0 9.57 10.00 9.57 9.74 9.49 9.55 Table VI : Total execution time -LRB- in seconds -RRB- required to sort a variety of double benchmarks on a 64-node IBM SP-2-WN . SP-2 ; results obtained from other platforms validate this claim as well . Because of this independence , the remainder of this section will only discuss the performance of our sample sort on the Worst-Load Regular benchmark -LSB- WR -RSB- . The results in Tables VII and VIII together with their graphs in Figure 1 examine the scalability of our sample sort as a function of machine size . Results are shown for the T3D , the SP-2-WN , and the CM-5 . Bearing in mind that these graphs are log-log plots , they show that , for a given input size n , the execution time scales inversely with the number of processors p for -LRB- p < = 64 -RRB- . While this is certainly the expectation of our analytical model for doubles , it might at first appear to exceed our prediction of an O ? np log p ? computational complexity for integers . However , the appearance of an inverse relationship is still quite reasonable when we note that , for values of p between 8 and 64 , log p varies by only a factor of two . Moreover , this O ? np log p ? complexity is entirely due to the merging in Step -LRB- 9 -RRB- , and in practice , Step -LRB- 9 -RRB- never accounts for more than 30 % of the observed execution time . Note that the complexity of Step -LRB- 9 -RRB- could be reduced to O ? np ? for integers using radix sort , but the resulting execution time would , in most cases , be slower . Regular Sorting of 8M Integers -LSB- WR -RSB- Number of Processors Machine 8 16 32 64 128 CRAY T3D 3.23 1.73 0.976 0.594 0.496 IBM SP2-WN 2.73 1.38 0.761 0.472 0.410 TMC CM-5 - 7.83 3.99 2.29 2.55 Table VII : Total execution time -LRB- in seconds -RRB- required to sort 8M integers on a variety of machines and processors using the -LSB- WR -RSB- benchmark . A hyphen indicates that particular platform was unavailable to us . However , the results in Tables VII and VIII together with their graphs in Figure 1 also show that for p greater than 64 , the inverse relationship between the execution time and the number of processors begins to deteriorate . Table IX explains these results with a step by step breakdown of the execution times reported for the sorting of integers on the T3D . Step -LRB- 1 -RRB- clearly displays the O ? np ? complexity expected for radix sort , and it dominates the total execution time for small values of p . The transpose operation in Step -LRB- 2 -RRB- displays the ? o + np oe ? complexity we originally suggested . The dependence Regular Sorting of 8M Doubles -LSB- WR -RSB- Number of Processors Machine 8 16 32 64 128 CRAY T3D 5.25 2.65 1.41 0.827 0.619 IBM SP2-WN 7.95 4.05 2.09 1.18 0.870 TMC CM-5 - - 6.89 4.39 4.24 Table VIII : Total execution time -LRB- in seconds -RRB- required to sort 8M doubles on a variety of machines and processors using the -LSB- WR -RSB- benchmark . A hyphen indicates that particular platform was unavailable to us . Figure 1 : Scalability of sorting integers and doubles with respect to machine size . of o on p simply becomes more pronounced as p increases and np decreases . Step -LRB- 3 -RRB- exhibits the O -LRB- sp -RRB- complexity we anticipated , since for 2b 12 log -LRB- n =p -RRB- c , s is halved every other time p is doubled . Steps -LRB- 6 -RRB- and -LRB- 9 -RRB- display the expected O ? p2 log p ? and O ?? np + ns ? log p ? ? ss O ?? np + pnp ? log p ? for s ss qnp ? complexity , respectively . Steps -LRB- 7 -RRB- and -LRB- 8 -RRB- exhibit the most complicated behavior . The reason for this is that in Step -LRB- 7 -RRB- , each processor must exchange p subsequences with every other processor and must include with each subsequence a record consisting of four integer values which will allow the unshuffling in Step -LRB- 8 -RRB- to be performed efficiently . Hence , the O ? np2 + nsp + 4p ? transpose block size in the case of 128 processors is nearly half that of the the case of 64 processors -LRB- 1280 vs. 2816 -RRB- . This , together with the fact that o increases as a function of p , explains why the time required for Step -LRB- 7 -RRB- actually increases for 128 processors . Step -LRB- 8 -RRB- would also be expected to Step by Step Breakdown of Sorting 8M Integers Number of Processors -LRB- Number of Samples -RRB- Step 8 -LRB- 1024 -RRB- 16 -LRB- 512 -RRB- 32 -LRB- 512 -RRB- 64 -LRB- 256 -RRB- 128 -LRB- 256 -RRB- 1 2.320220 1.172284 0.591670 0.299310 0.151576 2 0.132129 0.069106 0.045686 0.029076 0.019693 3 0.008468 0.010606 0.026364 0.026372 0.053686 4 0.000015 0.000019 0.000028 0.000047 0.000085 5 0.000052 0.000078 0.000082 0.000128 0.000226 6 0.000390 0.001303 0.004339 0.012499 0.028225 7 0.130839 0.070650 0.050185 0.039518 0.076284 8 0.148714 0.077050 0.042443 0.034429 0.059114 9 0.485934 0.332238 0.215449 0.152325 0.107410 Total 3.226760 1.733333 0.976246 0.593705 0.496300 Table IX : Time required -LRB- in seconds -RRB- for each step of sorting 8M integers on the Cray T3D using the -LSB- WR -RSB- benchmark . exhibit O ? np + ns ? ? ss O ? np + pnp ? for s ss qnp ? complexity . But the scheme chosen for unshuffling also involves an O -LRB- p -RRB- amount of overhead for each group of p subsequences to assess their relationship so that they can be efficiently unshuffled . For sufficiently large values of p , this overhead begins to dominate the complexity . While the data of Table IX was collected for sorting integers on the T3D , the data from the SP-2-WN and the T3D support the same analysis for sorting both integers and doubles . The graphs in Figure 2 examine the scalability of our regular sample sort as a function of keys per processor ? np ? , for differing numbers of processors . They show that for a fixed number of up to 64 processors there is an almost linear dependence between the execution time and np . While this is certainly the expectation of our analytic model for integers , it might at first appear to exceed our prediction of a O ? np log n ? computational complexity for floating point values . However , this appearance of a linear relationship is still quite reasonable when we consider that for the range of values shown log n differs by only a factor of 1:2 . For p > 64 , the relationship between the execution time and and np is no longer linear . But based on our discussion of the data in Table IX , for large p and relatively small n we would expect a sizeable contribution from those steps which exhibit O ? p2 log p ? , O ? np + pnp ? , and O ?? np + pnp ? log p ? complexity , which would explain this loss of linearity . Finally , the graphs in Figure 3 examine the relative costs of the nine steps in our regular sample sort algorithm . Results are shown for both a 64 node T3D and a 64 node SP-2-WN , using both the integer and the double versions of the -LSB- WR -RSB- benchmark . Notice that for n = 64M integers , the sequential sorting , unshuffling , and merging performed in Steps -LRB- 1 -RRB- , -LRB- 8 -RRB- , and -LRB- 9 -RRB- consume approximately 85 % of the execution time on the T3D and approximately 75 % of the execution time on the SP-2 . By contrast , the two transpose operations in Steps -LRB- 2 -RRB- and -LRB- 7 -RRB- together consume

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

parallel algorithms
sorting by regular sampling

generalized sorting
integer sorting
parallel performance