New Rules for Early Delivery in an Atomic Multicast System Theodore Johnson and Lionnel Maugis Dept. of CIS , University of Florida Gainesville , Fl 32611-2024 ted@cis.ufl.edu April 19 , 1994 Abstract An atomic multicast facility guarantees that a multicast message is delivered to the multicast group , and that every process receives messages in the same order . The availability of an atomic multicast facility greatly simplifies the implementation of many distributed system functions . Atomic multicast can be built on top of a reliable causal multicast by waiting until each processor in the group has multicast a message , then delivering the roots of the causal DAG . However , this approach makes the system run at the speed of the slowest processor . Performance can be improved by permitting an early delivery of messages . Dolev , Malki , and Kramer gave a protocol , ToTo , for early delivery atomic multicast built on a reliable causal multicast . In this paper , we show that the ToTo protocol is more restrictive than it needs to be , and present a simplification that improves performance . Next , we show how to extend the simplified ToTo protocol to permit early delivery in a wider range of operating conditions . Finally , we discuss a simple method to deliver some of the multicast messages even earlier . 1 Introduction Group communication protocols have been proposed to simplify the development of distributed applications . They guarantee reliable delivery and some defined order of messages for all recipients , even if the underlying communication subsystem is unreliable . In this paper , we do not consider synchronous systems , as they make strong assumptions about communication and processing delays -LSB- Cri90 -RSB- . Such systems are designed to cope with the worst possible situation , and may not take advantage of the average case . As processes replicate to provide fault-tolerance and availability , they need to cooperate when executing a global event -LCB- updating a replica for example . They generally agree on the construction of a global order aiming at mastering the uncertainties induced by asynchronism and failures . With this unique perception of the system 's evolution , all nodes can take consistent decisions , preventing a replica from diverging . Atomic multicast primitives provide this total ordering consensus , as the following properties are guaranteed : 1 . Atomicity . If a message is received by one member of the group , then it is received by all non-failed members . 2 . Order . The sequence of delivered messages is identical for all group members . 3 . Termination . Once a message has been successfully issued by a sender , eventually all group members will receive the message . Early distributed agreement protocols are based on variations of two or three phase commit -LSB- LG90 -RSB- , central control -LSB- BSS91 -RSB- or token ring -LSB- BG93 , AMMS +93 -RSB- . In addition to reliable and atomic multicast primitives , a membership facility provides virtually synchronous group of processes : all working processes receive the same set of messages between configuration changes . The communication topology is assumed to be fully connected and point to point . The channels are also reliable -LRB- eventual , exactly once delivery of uncorrupted messages -RRB- and first-in first-out . More recent protocols take advantage of the communication medium . Indeed , simulating reliable point to point links over an unreliable broadcast domain significantly wastes network bandwidth -LSB- Bra92 -RSB- . From a theoretical standpoint , the requirements for distributed agreement in a broadcast domain are given by Melliar-Smith , Moser and Agrawal -LSB- MMSA93 -RSB- . They proved that a necessary and sufficient condition for the existence of a deterministic consensus protocol is delivery of each broadcast message to at least d -LRB- n + k + 1 -RRB- = 2e process in an n-process system subject to k crash failures . In their broadcast model , if a message is delivered , it is delivered immediately and in order but not necessarily to all processes . In the Trans system -LSB- MSMA90 -RSB- , they designed a reliable atomic broadcast primitive for unreliable datagrams . Messages contain piggyback information to perform recovery and decide which process has received a particular message . A message mp sent by p contains two lists of message identifiers , ack -LRB- mp -RRB- and nack -LRB- mp -RRB- . All i in ack -LRB- mp -RRB- have been received by p -LRB- positive acknowledgments -RRB- , and all j in nack -LRB- mp -RRB- have been lost by p and need to be retransmitted -LRB- negative acknowledgments -RRB- . Using the transitive closure of these acknowledgments , they derived a partial order which can in turn be converted into a total order . Even if reliable sends and receives are achieved efficiently , the processing cost of the total ordering protocol is high : the delivery predicate is not built incrementally , the ordering has to take into account the recovery of lost messages . Dolev , Kramer and Malki refined this approach in the Transis system -LSB- DKM92 -RSB- . If i is in ack -LRB- mp -RRB- then p has received i and all j in ack ? -LRB- i -RRB- , where ack ? is the transitive closure of ack . Therefore , acknowledgments directly create a causal order , simplifying the total ordering -LRB- Toto -RRB- and group membership protocol . Both use this partial order and have a reasonable processing cost . Their membership algorithm has also the unusual capability of supporting more than one network partition -LSB- DMS93 -RSB- . As we studied the Toto protocol , we found that the delivery rules are somewhat redundant and could be improved to make the delivery even more aggressive . This is the goal of the next sections . 2 Early Delivery In this paper , we assume that we are building an atomic multicast facility on top of a reliable causal multicast facility . The reliable multicast facility provides a reliable group membership facility and the causality DAG . Reliable Group Membership : Reliable broadcast requires either that every working processor receive a message , or that none of the processors receive the message . A method for keeping track of the currently working processors is required , or otherwise the first failure prevents all future messages from being delivered . The current set of messages that receive the multicasts is the process group . The process group is modified by a configuration change . Every process sees a consistent set of configuration changes . When a processor fails , it is still desirable to deliver as many of the undelivered messages as possible . Birman -LSB- BSS91 -RSB- introduced the concept of virtual synchrony , which requires that if one processor delivers a message m before a configuration change , then all non-failed processors deliver m before the configuration change . The use of virtual synchrony gives a solid meaning to the delivery list . In addition , it permits the delivery of the pending messages , since the requirement is only that non-failed processors receive the message . Implementing reliable group membership and virtual synchrony requires a solution to the asynchronous agreement problem . Many solutions exist , and we assume that one of the solutions is available . We also assume that the group membership facility guarantees virtual synchrony . Causality DAG : The causality DAG is formed by the acknowledgments , ack , carried by the messages . The vertices are the messages , and -LRB- m1 ; m2 -RRB- is an edge if and only if message m1 acknowledges message m2 . Note that the DAG could be constructed using any causal multicast primitive . We assume that such DAG is available for use by the total ordering algorithm . Because of virtual synchrony , we only need to order the messages that are delivered between configuration changes . Because we have causal message delivery , messages are delivered to the processors in a mutually consistent order . In particular , the causality DAG is the same at all processors -LRB- although it is not necessarily revealed in the same order at all processors -RRB- . That is , if message m is added to the causality DAG Gp at processor p , then all messages that causally precede m have already been added to Gp . Therefore , if processor q has also added m to its causality DAG Gq , q has also added all messages that precede m to Gq . A simple algorithm for totally ordered delivery is to wait until there is a message in the causality DAG from every processor , then deliver the root messages in lexicographic order . While this algorithm provides total ordering at no additional cost in messages or acknowledgements , it delays delivery until every processor has sent a message . The delivery delay can be large , and it forces the system to operate at the speed of the slowest processor . Messages in the causality DAG can be delivered early -LRB- before the receipt of a message from every processor -RRB- if the messages satisfy a condition that will be detected by all processors , no matter in what order the causality DAG is revealed . At processor p , the total ordering protocol maintains the most recent portion of the causality DAG , G . A message is added to G when the causal multicast facility releases it to the total ordering protocol . When a message is delivered to the processor , it is removed from G . The messages that can be delivered are the roots of G . We call the roots the candidate messages . Early delivery is triggered when early delivery rule is satisfied . At this point , some number of candidates are eligible for early delivery , we call them source messages . The source messages can be delivered when the protocol is guaranteed that every other processor will deliver the same set of messages . Thus the set of source messages must be stable : Internal Stability : No non-source candidate message in G can become a source message . External Stability : No unseen message -LRB- not in G -RRB- can become a source message . If the current set of source messages is both internally and externally stable , they can be delivered . The trick is to devise a stability rule that works with the early delivery rule to guarantee correctness without requiring that a message be delivered from every candidate . 3 A Refinement of the ToTo Delivery Rule In this section , we present a simplification and improvement of the ToTo early delivery algorithm of Dolev , Kramer , and Malki . We generally follow their terminology , but modify it where convenient and to simplify the presentation . The ToTo protocol is built on top of a reliable causal broadcast , as described above . When a message can be causally delivered , the causality DAG G is augmented and the ToTo protocol is invoked . ToTo monitors G until it detects that some messages can be delivered . The delivery of messages creates a new activation of the ToTo protocol . The functions we define are to be evaluated on the current causality DAG when ToTo is invoked . In the following discussion , we define some functions that the early delivery protocol uses . The functions are evaluated on the current causality DAG G at processor p . If there can be confusion about which causality DAG on which processor the functions are evaluated , we annotate the functions . Otherwise , we suppress the extra notation for simplicity . Let P be the set of processors in the group . The first message in the causality DAG from a particular processor casts votes for the candidate messages that it causally follows . In addition , a candidate message votes for itself . Each candidate message , m , tracks the processors that voted for it in its vote vector , V V -LRB- m -RRB- . The ith component of the vote vector , V V -LRB- m -RRB- -LSB- i -RSB- is vote that processor i casts for m , and is defined by : V V -LRB- m -RRB- -LSB- i -RSB- = 1 If processor i votes for m If processor i votes ; but not for m ? if processor i has not cast a vote : We define the tail of a candidate message , m , to be the set of processors that sent a message that causally follow m. Then , N tail -LRB- m -RRB- = jtail -LRB- m -RRB- j . The tail of the causality DAG G is the set of all processors that have a message in G , and N tail -LRB- G -RRB- = jtail -LRB- G -RRB- j. Let u be the number of processors that have not voted yet -LRB- unseen votes -RRB- : u = jP ? ntail -LRB- G -RRB- j = jfj : V V -LRB- m1 -RRB- -LSB- j -RSB- = ? gj = n ? N tail -LRB- G -RRB- We denote the number of votes that a candidate message m receives to be nvt -LRB- m -RRB- = jfi : V V -LRB- m -RRB- -LSB- i -RSB- = 1gj Candidate messages are compared on the basis of the votes they receive . Let ? > = n = 2 be a threshold parameter . Let m1 and m2 be two candidate messages . We define a per-vote function , beats , and votes by beats -LRB- m1 ; m2 -RRB- -LSB- j -RSB- = 1 If V V -LRB- m1 -RRB- -LSB- j -RSB- = 1 and V V -LRB- m2 -RRB- -LSB- j -RSB- = Otherwise votes -LRB- m1 ; m2 -RRB- = jfj : beats -LRB- m1 ; m2 -RRB- -LSB- j -RSB- = 1gj Candidate message m1 wins over candidate message m2 if it beats m2 in more than ? votes . We define a function , W in as W in -LRB- m1 ; m2 -RRB- = 1 If votes -LRB- m1 ; m2 -RRB- > ? If votes -LRB- m2 ; m1 -RRB- > ? X Otherwise The value of W in is defined on the current causality DAG , G . We need a function that tells us about all possible future extensions of G , G0 . Thus , we use Future , which is all possible future values of W in : FutureG -LRB- m1 ; m2 -RRB- = fW inG0 -LRB- m1 ; m2 -RRB- in G0 extending Gg The only interesting ways that G0 can extend G is by specifying the votes of the processors that have no message in G. Thus , 1 2 Future -LRB- m1 ; m2 -RRB- if and only if the number of processors that vote for m1 but not m2 , plus the number of unseen votes -LRB- \* '' votes -RRB- is greater than ? . That is , 1 2 Future -LRB- m1 ; m2 -RRB- If votes -LRB- m1 ; m2 -RRB- + u > ? X 2 Future -LRB- m1 ; m2 -RRB- If X 2 W in -LRB- m1 ; m2 -RRB- 2 Future -LRB- m1 ; m2 -RRB- If 1 2 Future -LRB- m2 ; m1 -RRB- Note that 1 62 Future -LRB- m1 ; m2 -RRB- , votes -LRB- m1 ; m2 -RRB- + u < = ? , votes -LRB- m1 ; m2 -RRB- < = ? + N tail -LRB- G -RRB- ? n Future -LRB- m1 ; m2 -RRB- = f1g , W in -LRB- m1 ; m2 -RRB- = 1 Let MG be the set of candidate messages in the causality DAG G , The set of source messages SG is defined as i 2 SG , i 2 MG ^ 8j 2 MG ; 1 62 Future -LRB- j ; i -RRB- A message i is a source if it will never lose against any other candidate message j already in G -LRB- j can not beat i in more than ? votes in any possible extension of G -RRB- . The ToTo protocol is specified by its delivery rules : ToTo Delivery Rules 1 -RRB- -LRB- early delivery rule -RRB- if a -RRB- -LRB- internal stability -RRB- 8m 2 MGnSG ; 9m0 2 SG ; Future -LRB- m0 ; m -RRB- = f1g , and b -RRB- -LRB- external stability -RRB- 9s 2 SG ; nvt -LRB- s -RRB- > ? ^ 8s0 2 SG ; N tail -LRB- s0 -RRB- > = n ? ? . Then deliver SG . 2 -RRB- -LRB- default delivery rule -RRB- Else , if N tail -LRB- G -RRB- = n , Then deliver MG . The ToTo protocol can be improved and simplified by observing that part of rule 1b -RRB- is redundant . This observation leads to the Simplified ToTo -LRB- S-ToTo -RRB- protocol : Simplified ToTo Delivery Rules 1 -RRB- -LRB- early delivery rule -RRB- if a -RRB- -LRB- internal stability -RRB- 8m 2 MGnSG ; 9m0 2 SG ; Future -LRB- m0 ; m -RRB- = f1g . and b -RRB- -LRB- external stability -RRB- 9s 2 SG ; nvt -LRB- s -RRB- > ? , Then deliver SG . 2 -RRB- -LRB- default delivery rule -RRB- Else , if N tail -LRB- G -RRB- = n , Then deliver MG . SG is stable when a -RRB- all non-source candidate messages will not become sources in the current activation -LRB- if they lose once -RRB- and b -RRB- all unseen messages will not become sources -LRB- if a source message gets more than ? votes , they will lose against it -RRB- . The observation is that the first part of rule 1b -RRB- already guarantees external stability . As messages are inserted in a causal order , a processor p that already has a message in the graph will not vote for an unseen message . The first message from p , mp , votes for the candidate messages it causally follows which are already in G -LRB- or itself if it is a candidate message -RRB- . Suppose that the early delivery rule is satisfied . Then there exists a source s with at least ? + 1 votes . s can share these votes with other candidate messages in G , but all ? + 1 processors that have voted for s will not vote for an unseen message s0 . When G is extended by inserting s0 , if s0 is a candidate message , s0 will not become a source message . Indeed , s0 will lose against s , as a majority of processors will still vote for s and not for s0 : votes -LRB- s ; s0 -RRB- > ? . If s0 gather all unseen votes , nvt -LRB- s -RRB- + nvt -LRB- s0 -RRB- = n -RRB- nvt -LRB- s0 -RRB- < n ? ? . As ? > = n = 2 ; nvt -LRB- s0 -RRB- < ? . Therefore , 8i 2 SG ; votes -LRB- s0 ; i -RRB- < ? -LRB- s0 will not take any source message i out of SG -RRB- . The internal stability rule then guarantees that every process delivers the same set of messages on every activation . A non-source candidate message i will not become a source message if i loses against one candidate message j -LRB- if win -LRB- j ; i -RRB- = 1 in G , win -LRB- j ; i -RRB- = 1 in any extension of G -RRB- . We have presented an informal proof of correctness based on internal and external stability arguments while ignoring the interaction of the protocol with failures and group changes . These details are discussed in -LSB- DKM92 -RSB- . We note that ToTo delivery rule 1b -RRB- is only used to guarantee that when the delivery of messages occurs , the processor has received every source message . 4 Generalized ToTo The S-ToTo protocol can permit the delivery of a message after messages have been received from as few as n = 2 + 1 processors . Thus , the S-ToTo protocol will work well if all processors are connected by a bus . If the network consists of several LANs connected by gateways , it might become unlikely that any message will receive half of the votes . Thus , it would be useful to trigger early delivery by messages that receive only a large minority of the votes . The cost to the protocol is a requirement that more total votes be gathered , to ensure external stability . Again , we define our threshold to be ? , and we restrict 1 < ? < n . We need a definition of a source message that is simultaneously weak enough to allow the delivery of many source messages , but strong enough to quickly determine which candidates are non-source messages . We retain the meanings of W in and Future from the ToTo protocol , but we generalize the meaning of a source message as follows : i 2 SG , i 2 MG ^ -LSB- nvt -LRB- i -RRB- > ? _ 8j 2 MG ; 1 62 Future -LRB- j ; i -RRB- -RSB- It is necessary to allow a message with at least ? + 1 votes to become source message , because otherwise two messages with non-overlapping sets of ? + 1 votes may block each other from becoming sources . Note that the generalized definition of a source is the same as that for the ToTo protocol when ? > = n = 2 . With the generalized definition of a source the Generalized ToTo -LRB- G-ToTo -RRB- protocol is : Generalized ToTo Delivery Rules 1 -RRB- -LRB- early delivery rule -RRB- if a -RRB- -LRB- internal stability -RRB- 8i 2 MGnSG ; nvt -LRB- i -RRB- + u < = ? ^ 9j 2 SG ; Future -LRB- j ; i -RRB- = f1g and b -RRB- -LRB- external stability -RRB- N tail -LRB- G -RRB- > = n ? ? ^ 9s 2 SG ; nvt -LRB- s -RRB- > ? , Then deliver SG . 2 -RRB- -LRB- default delivery rule -RRB- Else , if N tail -LRB- G -RRB- = n , Then deliver MG . Note that when ? > = n = 2 , the G-ToTo protocol reduces to the S-ToTo protocol . The correctness of the G-ToTo protocol is a matter of showing that delivery occurs only when internal and external stability are achieved . Internal stability is explicitly tested . External stability is ensured because any possible source s must be in the causality DAG G when the delivery occurs . Suppose s is a source not in G when delivery occurs . But , s can not receive more than ? votes , because N tail -LRB- G -RRB- > = n ? ? . Since the early delivery rule was triggered , there is a source s0 with at least ? + 1 votes , so 1 2 Future -LRB- s0 ; s -RRB- . Therefore s can not be a source . 5 Early Delivery in an Activation We make one final observation to permit the delivery of some messages while the activation is being calculated -LRB- GL-Toto -RRB- . The sources messages are delivered in a deterministic order , by process id for example . If a message mA from processor A becomes a source and will not be taken out of the sources when further votes are gathered , then mA can be delivered even if the protocol has not reached a stable state . More specifically , if a prefix w of MG -LRB- ordered set -RRB- is stable , we can deliver all source messages in w. w is constructed iteratively . Each time the leftmost mi 2 MGnw is considered . If i > min -LRB- Pnntail -LRB- G -RRB- -RRB- , min -LRB- Pnntail -LRB- G -RRB- -RRB- has not voted yet and we have to stop the construction of w . If mi 2 SG , mi is added to w if no unseen message can win over mi : nvt -LRB- mi -RRB- > ? _ N tail -LRB- G -RRB- > = n ? ? , otherwise we stop . If mi 2 MGnSG , mi can be added to w if mi will not become a source message : nvt -LRB- mi -RRB- + u < = ? ^ 9mj 2 MG ; Future -LRB- mj ; mi -RRB- = f1g . Note that delivering early sources messages in w is valid even if the default delivery rule is applied -LRB- N tail -LRB- G -RRB- = n -RRB- . If the delivery can not occur before all votes are gathered , this means that either the set of sources needs to be extended , or that the unseen message is not a candidate message . In both cases , when N tail = n ; SG = MG . Here is a pseudo-code for such rules that is to be tested in the current activation , every time a new message is inserted in G : for i = min -LRB- P -RRB- to min -LRB- Pnntail -LRB- G -RRB- -RRB- ? 1 if mi is not marked if mi 2 SG if -LRB- nvt -LRB- mi -RRB- > ? _ N tail -LRB- G -RRB- > = n ? ? -RRB- mark mi ; deliver mi ; else stop ; else if -LRB- mi 2 MG -RRB- if -LRB- nvt -LRB- mi -RRB- + u < = ? ^ 9j 2 MG Future -LRB- j ; i -RRB- = f1g -RRB- / * mi is stable , continue * / else stop ; else / * mi 62 MG , continue * / ; else / * mi has already been delivered in the same activation , continue * / ; 6 Example I H JA C D E FB G The following example1 presents an activation of the protocol . Only the first message from a particular processor is shown . The causality DAG is depicted after the insertion of message H. n = 12 ; ? = 4 ; N tail -LRB- G -RRB- = 10 ; u = N tail -LRB- G -RRB- ? n = 2 ; MG = fA ; B ; F ; I ; Jg mi A B F I J nvt -LRB- mi -RRB- jfAgj = 1 jfB ; C ; D ; E ; Ggj = 5 jfC ; D ; E ; F gj = 4 jfH ; Igj = 1 jfJgj = 1 votes -LRB- row ; col -RRB- A B F I J A ? jfAgj = 1 1 1 1 B jfB ; C ; D ; E ; Ggj = 5 ? jfB ; Ggj = 2 5 5 F jfC ; D ; E ; F gj = 4 jfF gj = 1 ? 4 4 I jfH ; Igj = 2 2 2 ? 2 J jfJgj = 1 1 1 1 ? SG = fB ; Fg is computed as follows : ffl A 62 SG as nvt -LRB- A -RRB- = 1 < = ? ^ 9B 2 MG ; votes -LRB- B ; A -RRB- + u = 5 + 2 > ? ffl B 2 SG as nvt -LRB- B -RRB- > ? ffl F 2 SG as votes -LRB- A ; F -RRB- + u = 1 + 2 < = ? ^ votes -LRB- B ; F -RRB- + u = 4 < = ? ^ votes -LRB- I ; F -RRB- + u = 4 < = ? ^ votes -LRB- J ; F -RRB- + u = 3 < = ? ffl I 62 SG as nvt -LRB- I -RRB- = 2 < = ? ^ 9B 2 MG ; votes -LRB- B ; I -RRB- + u = 5 + 2 > ? ffl J 62 SG as nvt -LRB- J -RRB- = 1 < = ? ^ 9B 2 MG ; votes -LRB- B ; J -RRB- + u = 5 + 2 > ? Sp is stable and can be delivered : 1Neither Total -LSB- MSMA90 -RSB- nor ToTo -LSB- DKM92 -RSB- are able to elect messages with such DAG , as they require a candudate set to receive more than n = 2 votes to be elected . 1 . MGnSG = fA ; I ; Jg nvt -LRB- A -RRB- + u = 3 < = ? ^ 9B 2 SG ; votes -LRB- B ; A -RRB- = 5 > ? ^ nvt -LRB- I -RRB- + u = 4 < = ? ^ 9B 2 SG ; votes -LRB- B ; I -RRB- = 5 ^ nvt -LRB- J -RRB- + u = 3 < = ? ^ 9B 2 SG ; votes -LRB- B ; J -RRB- = 5 2 . N tail -LRB- G -RRB- = 10 > = n ? ? ^ 9B 2 SG ; nvt -LRB- B -RRB- > ? Now suppose that H has not been inserted yet . The functions are modified as follows : N tail -LRB- G -RRB- = 9 ; u = 3 ; nvt -LRB- I -RRB- = 1 ; votes -LRB- I ; A -RRB- = votes -LRB- I ; B -RRB- = votes -LRB- I ; F -RRB- = votes -LRB- I ; J -RRB- = 1 SG = fBg as ffl 8i 2 fA ; I ; Jg ; i 62 SG as nvt -LRB- i -RRB- < = ? ^ 9B 2 MG ; votes -LRB- B ; i -RRB- + u = 5 + 3 > ? ffl B 2 SG as nvt -LRB- B -RRB- > ? ffl F 62 SG as nvt -LRB- F -RRB- < = ? ^ 9B 2 MG ; votes -LRB- B ; F -RRB- + u = 2 + 3 > ? Furthermore , SG is not internally stable , thus can can not be delivered : 9F 2 MGnSG = fA ; F ; I ; Jg ; nvt -LRB- F -RRB- + u = 4 + 3 > ? Even if the protocol has not reached a stable state , we can safely deliver B . The prefix fA ; Bg of MG is stable . 1 . A 2 MGnSG ^ nvt -LRB- A -RRB- + u = 4 < = ? ^ 9B 2 MG ; votes -LRB- B ; A -RRB- = 5 > ? : A will not become a source message . 2 . B 2 SG ^ nvt -LRB- B -RRB- > ? : B will stay a source message . 7 Improvements The family of total ordering protocols presented in this paper is geared towards reducing the average latency of agreed message delivery . We measure this latency as being the number of processors one need to hear from before delivering a message -LRB- N tail -LRB- G -RRB- -RRB- . Each time a message mi is agreed and delivered , we record also its number of votes nvt -LRB- mi -RRB- . Our initial study on performance shows that the improvements are quite noticeable . We point out that the modifications needed for S-Toto , G-Toto and GL-Toto are straightforward and do not generate additional processing cost . Experimental results with our implementation of the protocols and simple simulations of a wider range of causal DAG showed that : 1 . As much as 10 % of the messages delivered early by S-Toto are not delivered by Toto . 2 . As much as 5 % of all messages can be delivered early with the lexicographic order . 3 . GL-Toto performs better than Toto with wide causal graphs . The latter uses the default delivery rule -LRB- N tail -LRB- G -RRB- = n -RRB- most of the time , while GL-Toto can delivery messages when it receives more than n ?? votes . 4 . We can order up to 400 messages per second on a 1MBit = sec ethernet , on a n = 8 processor system . With ? = n = 2 , the average N tail -LRB- G -RRB- is 5:12 . In our testbed , performance depends heavily on the communication patterns , if there is a lot of traffic on the net for example . Furthermore , the above results do not take into account process failure or message recovery . 8 Conclusion In a broadcast domain , a reliable causal multicast facility can be efficiently implemented without sending additional messages . Using this causal order and a special voting scheme , we have presented a family of protocols for delivering group messages in a unique order -LRB- again , without additional messages -RRB- . The protocol reduces delivery latency by inserting messages in the total order as soon as they get elected , even if votes from all members have not been gathered -LRB- yet -RRB- . The delivery rules are parameterized to cope with a wide class of communication patterns . The set of elected messages is made as large as possible , is easy to compute , but depends on a majority threshold parameter ? . Small values of ? improve performance for wide causal DAG , while ? = n = 2 seems to be a good choice for narrow graphs . References -LSB- AMMS +93 -RSB- Y. Amir , L. E. Moser , P.M. Melliar-Smith , V. Agrawal , and P. Ciarfella . Fast message ordering and membership using a logical taken passing ring . Intl. . Conference on Distributed Computing Systems , May 1993 . -LSB- BG93 -RSB- T. Becker and K. Grieger . An efficient atomic muticast protocol for client server models . IEEE International parallel processing symposium , 1993 . -LSB- Bra92 -RSB- R. Braden . Extending fTCPg for transactions . Internet Network Group -LCB- Request for Comments 1379 , 1992 . -LSB- BSS91 -RSB- K. Birman , A. Schiper , and P. Stephenson . Lightweight causal and atomic group multicast . ACM Transactions on computer systems , 9 -LRB- 3 -RRB- , 1991 . -LSB- Cri90 -RSB- F. Cristian . Synchronous atomic broadcast for redundant broadcast channels . The Journal of Real-Time systems , 2 , 1990 . -LSB- DKM92 -RSB- D. Dolev , S. Kramer , and D. Malki . Total ordering of messages in broadcast domains . Technical report , The Hebrew University of Jerusalem , November 1992 . -LSB- DMS93 -RSB- D. Dolev , D. Malki , and Ray Strong . An asynchronous membership protocol that tolerates partitions . Technical report , The Hebrew University of Jerusalem , November 1993 . -LSB- LG90 -RSB- S. Luan and V. Gligor . A fault tolerant protocol for atomic broadcast . IEEE Transactions on Parallel and Distributed Systems , 1 -LRB- 3 -RRB- , 1990 . -LSB- MMSA93 -RSB- L. E. Moser , P. M. Melliar-Smith , and V. Agrawala . Necessary and sufficient conditions for broadcast consensus protocols . Journal of Distributed Computing , 7 -LRB- 2 -RRB- , December 1993 . -LSB- MSMA90 -RSB- P.M. Melliar-Smith , L. Moser , and V. Agrawal . Broadcast protocols for distributed systems . IEEE transactions on parallel and distributde systems , 1 -LRB- 1 -RRB- , 1990 .

O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

atomic broadcast
causal broadcast
early delivery

asynchronous system
distributed protocol


High

Performance

Geographic

Information

Systems

Experiences

with

a

Shared

Address

Space

Architecture

Shashi

Shekhar

Sivakumar

Ravada

Vipin

Kumar

Douglas

Chubb

Greg

Turner

Abstract

Several

emerging

visualization

applications

such

as

flight

simulators

distributed

interactive

simulation

-LRB-

DIS

-RRB-

and

virtual

reality

are

using

geographic

information

systems

-LRB-

GISs

-RRB-

for

high-fidelity

representation

of

actual

terrains

These

applications

impose

stringent

performance

and

responsetime

restrictions

which

can

be

met

via

parallelization

of

the

GIS

Currently

we

are

developing

a

high

performance

GIS

on

a

Shared

Address

Space

Architecture

-LRB-

SASA

-RRB-

namely

Silicon

Graphics

Challenge

as

a

part

of

the

virtual

GIS

project

of

the

Army

Research

Laboratories

Our

experience

shows

that

data-parallel

message-passing

algorithms

can

be

fairly

effective

for

implementing

high

performance

GISs

on

SASAs

Our

experimental

results

with

the

GIS-rangequery

operation

show

that

data-partitioning

is

an

effective

approach

towards

achieving

high

performance

in

GIS

As

partitioning

extended

spatial

objects

is

difficult

special

techniques

such

as

systematic

declustering

are

needed

to

parallelize

the

range-query

operation

Systematic

declustering

methods

outperform

random

declustering

methods

for

range

query

problems

Static

load-balancing

methods

with

systematic

declustering

even

outperform

random

declustering

coupled

with

dynamic

load-balancing

-LRB-

DLB

-RRB-

In

addition

the

performance

of

DLB

methods

can

be

improved

by

using

the

declustering

methods

for

determining

the

subsets

of

polygons

to

be

transferred

during

run-time

Author

Address

E-mail

Phone/Fax

Shashi

Shekhar

Sivakumar

Ravada

Vipin

Kumar

Department

of

Computer

Science

University

of

Minnesota

EE/CS

Union

St.

SE

Minneapolis

MN

shekhar@cs.umn.edu

siva@cs.umn.edu

kumar@cs.umn.edu

Douglas

Chubb

Research

and

Technology

Division

U.S.

Army

CECOM

RDEC

IEWD

Vint

Hill

Farms

Station

Warrenton

VA

Greg

Turner

Army

Research

Laboratory

Adelphi

MD

This

work

is

sponsored

in

part

by

the

Army

High

Performance

Computing

Research

Center

under

the

auspices

of

the

Department

of

the

Army

Army

Research

Laboratory

cooperative

agreement

number

contract

number

the

content

of

which

does

not

necessarily

reflect

the

position

or

the

policy

of

the

government

and

no

official

endorsement

should

be

inferred

This

work

is

also

supported

in

part

by

NSF

grant

Introduction

A

high

performance

geographic

information

system

-LRB-

HPGIS

-RRB-

is

a

central

component

of

many

interactive

applications

like

real-time

terrain

visualization

situation

assessment

and

spatial

decision-making

The

geographic

information

system

-LRB-

GIS

-RRB-

-LSB-

-RSB-

often

contains

spatial

data

-LRB-

geometric

and

feature

data

-RRB-

represented

as

large

sets

of

points

chains

of

line-segments

and

polygons

In

practice

the

spatial

data

sets

required

to

model

geographic

applications

are

usually

very

large

-LRB-

MBytes

-RRB-

Since

interactive

and

real-time

applications

require

short

response

times

the

GIS

has

to

process

these

large

data

sets

in

a

very

short

time

The

existing

sequential

methods

for

supporting

the

GIS

operations

do

not

meet

the

real-time

requirements

imposed

by

many

interactive

applications

Parallelization

of

GIS

can

meet

the

high

performance

requirements

of

several

HPGIS

applications

In

this

paper

we

focus

on

parallelizing

the

range-query

operation

which

is

important

in

many

GIS

applications

on

a

shared

address-space

architecture

-LRB-

SASA

-RRB-

We

experimentally

show

that

data

partitioning

using

systematic

declustering

is

effective

for

achieving

good

performance

in

many

GIS

applications

Application

Domain

Real-Time

Terrain

Visualization

A

real-time

terrain-visualization

-LSB-

-RSB-

system

-LRB-

e.g.

a

distributed

interactive

simulation

system

-LSB-

-RSB-

-RRB-

is

a

virtual

environment

-LSB-

-RSB-

which

lets

the

users

navigate

and

interact

with

a

three-dimensional

computergenerated

geographic

environment

in

real-time

This

type

of

system

has

three

major

components

interaction

graphics

and

GIS

Figure

shows

the

different

components

of

a

terrain-visualization

system

for

a

typical

flight

simulator

The

HPGIS

component

of

the

system

contains

a

secondary

storage

unit

for

storing

the

entire

geographic

database

and

a

main

memory

for

storing

the

data

related

to

the

current

location

of

the

simulator

The

graphics

engine

receives

the

spatial

data

from

the

HPGIS

component

and

transforms

this

data

into

objects

which

are

then

sent

to

the

display

unit

As

the

user

moves

over

the

terrain

the

part

of

the

map

that

is

visible

to

the

user

changes

over

time

and

the

graphics

engine

has

to

be

fed

with

the

visible

subset

of

spatial

objects

for

a

given

location

and

user

's

viewport

The

graphics

engine

transforms

the

user

's

viewport

into

a

range

query

and

sends

the

query

to

the

HPGIS

unit

The

HPGIS

unit

retrieves

the

geographic

description

of

the

area

corresponding

to

the

range

query

and

sends

the

description

back

to

the

graphics

engine

For

example

Figure

shows

a

polygonal

map

representing

vegetation

types

at

a

given

location

together

with

a

range

query

Each

polygon

in

the

map

represents

a

land

parcel

with

a

given

type

of

vegetation

Usually

a

map

projection

is

used

to

compute

a

polygonal

representation

of

the

real

world

phenomena

which

are

measured

in

latitude

and

longitude

These

polygons

are

shown

with

dotted

lines

The

range

query

is

represented

by

the

rectangle

and

the

result

of

the

range

query

is

shown

in

solid

lines

The

HPGIS

unit

retrieves

the

spatial

data

from

the

main

memory

computes

its

geometric

intersection

with

the

current

viewport

of

the

user

and

sends

the

results

back

to

the

graphics

engine

The

frequency

of

this

operation

depends

on

the

speed

at

which

the

user

is

moving

over

the

terrain

For

example

in

the

terrain

visualization

of

a

flight

simulator

a

new

range

query

may

be

generated

twice

a

second

which

leaves

less

than

half

a

second

for

the

intersection

computation

A

typical

map

used

in

this

application

contains

tens

of

thousands

of

polygons

-LRB-

i.e.

millions

of

edges

-RRB-

and

the

range-query

size

can

be

of

the

total

map

This

requires

accessing

millions

of

edges

and

computing

millions

of

intersection-points

in

less

than

half

a

second

In

order

to

meet

such

response-time

constraints

a

HPGIS

often

caches

a

subset

of

spatial

data

in

main

memory

The

main-memory

database

may

in

turn

query

the

secondary-storage

database

to

get

a

subset

of

data

to

be

cached

The

frequency

of

this

operation

should

be

very

low

for

the

buffering

to

be

effective

Graphics

Analysis

Engine

Main-Memory

Database

Terrain

Secondary

Database

Storage

Terrain

Display

range-query

set

of

polygons

set

of

polygons

view

feed

back

High

Performance

GIS

Component

storage

secondary

range-query

Figure

Components

of

the

Terrain-Visualization

system

with

details

of

a

range

query

Problem

Formulation

The

range-query

problem

for

the

GIS

can

be

stated

as

follows

Given

a

rectangular

query

box

B

and

a

set

SP

of

extended

spatial

objects

-LRB-

e.g

polygons

chains

of

line

segments

-RRB-

the

result

of

a

range

query

over

SP

is

given

by

the

set

fxjx

Pi

B

SP

g

where

gives

the

geometric

intersection

of

two

extended

objects

We

call

this

problem

the

GIS-range-query

problem

The

computational

complexity

of

this

problem

for

a

given

polygon

is

proportional

to

the

number

of

edges

of

that

polygon

Note

that

this

problem

is

different

from

the

traditional

range

query

where

the

objects

in

the

given

range

are

retrieved

from

the

secondary

memory

-LRB-

disk

-RRB-

to

main

memory

without

clipping

the

objects

but

it

is

similar

to

the

polygon-clipping

problem

in

computer

graphics

In

this

paper

we

focus

on

parallelizing

the

GIS-range-query

problem

on

a

shared

address-space

architecture

-LRB-

SASA

-RRB-

Scope

Alternative

techniques

like

polygon

simplification

via

culling

-LSB-

-RSB-

and

incremental

range

queries

-LSB-

-RSB-

can

be

used

to

further

improve

the

performance

of

range

query

processing

Parallelization

of

the

GIS-rangequery

problem

is

orthogonal

to

these

alternatives

since

they

can

be

used

in

addition

to

the

parallelization

techniques

In

this

paper

we

only

focus

on

the

parallelization

issues

of

this

problem

The

parallel

formulations

discussed

in

this

paper

do

not

strictly

require

a

shared

address-space

architecture

However

the

availability

of

SASA

eases

the

implementation

of

the

dynamic

load-balancing

routines

as

discussed

in

section

We

also

focus

on

a

data-partitioning

approach

to

parallelizing

the

GIS-range-query

The

simple

approach

of

dividing

the

data

statically

between

processors

without

using

any

run-time

work-transfers

is

referred

to

as

the

static

load-balancing

-LRB-

SLB

-RRB-

approach

The

approach

of

using

work-transfers

during

run-time

to

improve

the

load-balance

is

referred

to

as

dynamic

load-balancing

-LRB-

DLB

-RRB-

Parallel

Formulation

for

the

GIS-range-query

The

GIS-range-query

problem

has

three

main

components

-LRB-

i

-RRB-

approximate

filtering

at

the

polygon

level

-LRB-

ii

-RRB-

intersection

computation

and

-LRB-

iii

-RRB-

polygonization

of

the

result

-LRB-

See

-LSB-

-RSB-

for

a

detailed

discussion

of

a

sequential

algorithm

-RRB-

A

search

structure

is

used

to

filter

out

many

non-interesting

polygons

from

the

set

of

input

polygons

The

query

box

is

then

intersected

with

each

of

the

remaining

polygons

and

the

output

is

obtained

as

a

set

of

polygons

by

polygonizing

the

results

of

the

intersection

computation

The

GIS-range-query

operation

can

be

parallelized

either

by

function-partitioning

-LSB-

-RSB-

or

by

datapartitioning

-LSB-

-RSB-

Function-Partitioning

uses

specialized

data

structures

and

algorithms

which

may

be

different

from

their

sequential

counterparts

A

data-partitioning

technique

divides

the

data

among

different

processors

and

independently

executes

the

sequential

algorithm

on

each

processor

Data-Partitioning

in

turn

is

achieved

by

declustering

-LSB-

-RSB-

the

spatial

data

If

the

static

declustering

methods

fail

to

equally

distribute

the

load

among

different

processors

the

load-balance

may

be

improved

by

redistributing

parts

of

the

data

to

idle

processors

using

dynamic

load-balancing

-LRB-

DLB

-RRB-

techniques

In

this

paper

we

focus

on

parallelizing

the

GIS-range-query

problem

using

the

data-partitioning

approach

Figure

shows

the

different

stages

of

the

parallel

formulation

In

the

following

discussion

let

P

be

the

number

of

processors

used

in

the

system

Initially

all

the

data

is

partitioned

into

two

sets

local

data

set

Sa

and

shared

data

set

Sb

The

data

in

Sa

is

statically

declustered

into

P

sets

Sia

for

i

-LRB-

P

-RRB-

and

Sia

is

mapped

to

the

local

address

space

of

processor

Pi

The

purpose

of

local

data

set

Sa

is

to

reduce

the

synchronization

overhead

of

accessing

the

data

The

set

Sb

is

shared

among

the

processors

for

dynamic

load-balancing

-LRB-

DLB

-RRB-

Approximate

filtering

Intersection

computation

polygonization

Approximate

filtering

Intersection

computation

polygonization

partition

static

get

next

range

query

DLB

DLB

S

Y

N

C

Figure

Different

stages

of

the

parallel

formulation

When

a

bounding

box

for

the

next

range

query

is

received

each

processor

Pi

performs

the

approximate

polygon-level

filtering

retrieves

the

candidate

polygons

from

its

local

data

set

Sia

and

starts

work

on

intersection

computation

and

polygonization

When

a

processor

Pi

finishes

work

on

the

local

data

it

goes

into

the

DLB

mode

to

process

the

polygons

from

Sb

Static

Declustering

of

Spatial

data

The

goal

of

declustering

is

to

partition

the

data

so

that

each

partition

imposes

approximately

the

same

load

for

any

range-query

Intuitively

the

polygons

close

to

each

other

should

be

scattered

among

different

processors

such

that

for

each

range-query

every

processor

has

an

equal

amount

of

work

Polygonal

maps

can

be

declustered

at

the

polygonal

level

or

at

the

sub-polygonal

level

We

focus

only

on

polygon-level

declustering

to

avoid

the

additional

work

of

merging

clipped

sub-polygons

Optimal

declustering

of

extended

spatial

data

-LRB-

like

polygons

-RRB-

is

difficult

to

achieve

due

to

the

nonuniform

distribution

and

variable

sizes

of

polygons

In

addition

the

load

imposed

by

a

polygon

for

each

range

query

depends

on

the

size

and

location

of

the

query

Since

the

location

of

the

query

is

not

known

a

priori

it

is

hard

to

develop

a

declustering

strategy

that

is

optimal

for

all

range

queries

As

the

declustering

problem

is

NP-Hard

-LSB-

-RSB-

heuristic

methods

are

used

for

declustering

spatial

data

Random

partitioning

local

load-balance

and

similarity-graph-based

methods

are

three

popular

algorithms

for

declustering

spatial

data

Intuitively

a

local

load-balance

method

tries

to

balance

the

work-load

at

each

processor

for

a

given

range

query

A

similarity

based

declustering

method

tries

to

balance

the

work-load

at

each

processor

over

a

representative

set

of

multiple

range

queries

Refer

to

-LSB-

-RSB-

for

a

detailed

discussion

of

the

declustering

algorithms

Dynamic

Load-Balancing

-LRB-

DLB

-RRB-

Techniques

If

static

declustering

methods

fail

to

equally

distribute

the

load

among

different

processors

the

loadbalance

may

be

improved

by

transferring

some

spatial

objects

to

idle

processors

using

dynamic

loadbalancing

techniques

A

typical

dynamic

load-balancing

technique

-LSB-

-RSB-

for

a

shared

memory

computer

addresses

two

issues

-LRB-

i

-RRB-

how

does

an

idle

processor

fetch

more

work

and

-LRB-

ii

-RRB-

what

is

the

granularity

of

the

work

transfer

and

the

data-partitioning

method

How

Does

an

Idle

Processor

Fetch

More

Work

In

SMPs

a

global

work-pool

is

often

maintained

in

the

shared

memory

An

idle

processor

accesses

this

pool

to

fetch

more

work

We

refer

to

this

method

as

the

Pool-Based

Method

-LRB-

PBM

-RRB-

Synchronization

may

be

necessary

to

access

this

pool

In

addition

the

structure

of

the

GIS-range-query

problem

imposes

a

limit

on

the

amount

of

work

that

can

be

kept

in

the

shared

pool

Since

the

filtering

step

is

not

easily

parallelizable

a

leader

processor

takes

this

responsibility

for

the

shared

pool

But

if

the

leader

processor

has

to

manage

too

big

a

pool

some

of

the

other

processors

might

finish

processing

their

local

data

before

the

leader

processor

can

perform

the

approximate

filtering

step

on

the

shared

pool

of

data

This

processing

at

the

leader

processor

forces

the

other

processors

to

wait

for

the

leader

processor

to

finish

the

filtering

step

on

pool

data

On

the

other

hand

too

small

a

pool

can

also

lead

to

processor

idling

due

to

the

static

load-imbalance

in

the

rest

of

the

data

These

two

situations

are

shown

in

Figure

IDLE

Process

DLB

IDLE

Process

DLB

Process

Process

DLB

IDLEApprxFil

-LRB-

Pool

-RRB-

Process

DLB

ApprxFil

-LRB-

Pool

-RRB-

Process

DLB

Pool

too

small

Pool

too

large

IDLE

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

Too

small

or

too

large

a

pool

may

result

in

processor

idling

Granularity

of

Transfers

and

Data-Partitioning

Method

Several

strategies

like

self-scheduling

factoring

scheduling

and

chunk

scheduling

-LSB-

-RSB-

exist

for

determining

the

amount

of

work

to

be

fetched

during

DLB

The

cost

of

synchronization

affects

the

performance

of

these

scheduling

algorithms

as

several

processors

have

to

share

the

knowledge

about

the

remaining

work

Small

chunks

can

be

used

when

the

synchronization

cost

is

low

In

the

case

of

chunks

of

more

than

one

object

it

is

desirable

to

keep

a

comparable

amount

of

work

in

each

chunk

so

that

the

load-imbalance

can

be

kept

low

Since

accurate

work

estimation

is

very

expensive

in

the

case

of

extended

spatial

data

objects

the

simple

scheduling

strategies

are

not

suitable

for

GIS

applications

However

if

work

can

be

divided

into

chunks

of

equal

size

any

of

the

scheduling

strategies

can

be

applied

which

treat

each

chunk

as

the

unit

of

work

transfer

We

note

that

this

problem

of

dividing

work

into

chunks

of

equal

size

is

similar

to

the

declustering

problem

We

hypothesize

that

the

load-balance

achieved

by

any

DLB

method

can

be

improved

by

using

a

systematic

declustering

method

for

dividing

the

work

into

chunks

Since

the

declustering

operation

is

expensive

to

perform

during

query

processing

this

chunking

can

be

done

during

the

loading

of

new

data

set

into

main

memory

As

this

loading

operation

is

performed

relatively

infrequently

-LRB-

less

than

once

in

minutes

-RRB-

there

will

be

sufficient

time

to

perform

the

declustering

operation

without

interrupting

the

actual

range-query

operation

Experimental

Evaluation

We

experimentally

evaluate

the

proposed

parallel

formulation

on

a

bus-based

SASA

with

processors

and

of

main

memory

-LRB-

SGI

Challenge

-RRB-

Each

node

of

this

machine

is

a

MIPS

processor

running

at

We

use

spatial

vector

data

for

Killeen

Texas

for

this

experimental

study

This

data

has

seven

themes

representing

the

attributes

slope

vegetation

surface

material

hydrology

etc.

We

used

the

slope

attribute-data

map

with

polygons

and

edges

as

a

base

map

in

our

experiments

-LRB-

this

is

denoted

by

map

-RRB-

In

the

rest

of

the

discussion

a

kX

map

denotes

a

map

that

is

k

times

bigger

than

the

map

Each

polygon

is

represented

as

a

sequence

of

edges

where

each

edge

is

a

pair

of

coordinate

points

In

addition

a

search

structure

-LSB-

-RSB-

is

used

to

quickly

identify

the

relevant

edges

of

a

polygon

for

a

given

range

query

Each

range

query

is

chosen

to

be

about

of

the

total

area

of

the

map

In

our

experiments

we

only

measure

and

analyze

the

cost

per

range-query

and

exclude

any

preprocessing

cost

This

preprocessing

cost

includes

the

cost

of

loading

the

data

into

main

memory

and

the

cost

of

declustering

the

data

among

different

processors

Note

that

this

preprocessing

cost

is

paid

only

once

for

each

data

set

and

that

this

operation

is

performed

less

than

once

in

minutes

The

experimental

results

are

compared

in

terms

of

the

speedups

achieved

by

different

methods

We

computed

the

speedup

S

for

P

processors

as

Tseq

TP

where

TP

is

the

parallel

run-time

on

P

processors

and

Tseq

is

the

sequential

runtime

on

a

single

processor

of

the

same

system

Note

that

there

are

several

parameters

involved

in

evaluating

an

algorithm

for

the

GIS-range-query

problem

Here

we

present

an

evaluation

of

only

those

parameters

which

are

interesting

for

SASAs

For

a

detailed

study

of

other

parameters

see

-LSB-

-RSB-

Speedup

Polygons

per

Chunk

P

P

P

Figure

Speedups

for

different

granularities

of

work

transfers

for

the

GRR

method

with

Map

Evaluation

of

the

Granularity

of

Work-Allocation

in

DLB

We

compare

the

effect

of

different

chunk

sizes

with

chunks

containing

to

polygons

using

PBM

for

DLB

and

the

similarity-graph

method

for

declustering

The

experiment

is

conducted

using

the

map

for

P

to

with

a

pool

Figure

shows

the

results

of

this

experiment

The

main

trends

observed

from

this

graph

are

-LRB-

i

-RRB-

that

chunk-size

affects

speedups

and

-LRB-

ii

-RRB-

that

the

chunk-size

providing

the

best

speedups

goes

up

with

an

increasing

number

of

processors

There

are

two

extremes

for

dividing

N

polygons

into

different

chunk

sizes

-LRB-

i

-RRB-

chunks

of

single

polygons

and

-LRB-

ii

-RRB-

chunks

of

N

=P

polygons

In

the

first

case

synchronization

cost

and

communication

bottleneck

due

to

limited

bus

bandwidth

limit

the

speedup

achieved

and

in

the

second

case

static

load

imbalance

limits

the

speedup

achieved

Hence

the

peak

speedup

is

usually

achieved

somewhere

in

between

these

two

extremes

and

the

speedup

curves

in

Figure

reflect

this

effect

In

addition

as

the

number

of

processors

increase

the

load

imbalance

increases

increasing

the

number

of

chunks

transferred

at

runtime

Hence

the

optimal

chunk

size

for

maximum

speedup

is

sensitive

to

the

number

of

processors

Declustering

for

DLB

Methods

In

this

experiment

the

effect

of

using

declustering

methods

for

chunking

inside

a

fixed

DLB

is

evaluated

We

used

PBM

as

the

DLB

method

and

compared

random

similarity-graph

and

LLB

methods

of

declustering

The

dynamic

data

Sb

is

declustered

with

fixed-size

chunks

-LRB-

polygons

per

chunk

-RRB-

Figure

shows

the

experimental

results

The

x-axis

gives

the

number

of

processors

and

the

y-axis

gives

the

average

speedup

over

queries

From

this

data

it

is

clear

that

random

declustering

of

data

is

not

as

effective

as

systematic

declustering

for

achieving

a

good

load-balance

in

the

GIS-range-query

problem

This

shows

that

systematic

declustering

of

data

improves

the

load-balance

Also

note

that

SLB

with

systematic

declustering

and

without

any

DLB

is

better

than

SLB

coupled

with

DLB

using

random

declustering

-LRB-

as

in

Figure

-RRB-

Speedup

Number

of

Processors

sim

LLB

RAN

Figure

Speedups

for

different

declustering

methods

for

the

PBM

method

with

Map

Effect

of

Pool

Size

for

PBM

We

evaluate

the

effect

of

the

pool

size

using

PBM

as

the

DLB

method

The

number

of

processors

is

varied

from

to

and

the

map

is

varied

from

map

to

map

The

pool

size

is

varied

from

to

of

the

total

data

Note

that

a

pool

refers

to

static

declustering

with

no

DLB

Figure

shows

the

results

of

this

experiment

The

x-axis

gives

the

size

of

the

pool

as

a

percentage

of

the

total

data

and

the

y-axis

gives

the

average

speedups

over

range

queries

Speedup

Pool

Size

Map

Map

Map

Map

Map

Speedup

Pool

Size

P

P

P

-LRB-

a

-RRB-

P

-LRB-

b

-RRB-

Map

Figure

Speedups

for

different

pool

sizes

for

Pool-Based

method

The

speedups

increase

as

we

increase

the

pool

size

up

to

a

point

and

then

they

start

decreasing

The

initial

increase

in

speedups

may

be

due

to

the

improved

load-balance

The

decrease

in

speedup

after

achieving

a

maximum

value

is

due

to

the

non-parallelizable

overhead

of

the

approximate

filtering

This

result

is

consistent

with

observations

made

in

Figure

The

pool

size

as

a

fraction

of

the

total

work

that

results

in

the

best

speedup

decreases

with

increasing

map

size

This

may

be

due

to

the

improvement

in

static

load-balance

due

to

the

increased

total

work

Parallel

Programming

Models

for

GIS-range-query

Problem

A

parallel

implementation

for

the

range-query

problem

can

be

developed

using

either

a

PRAM

algorithm

or

a

message-passing

algorithm

PRAM

algorithms

often

assume

that

the

cost

of

synchronization

and

processor

interaction

are

negligible

Thus

PRAM

algorithms

often

result

in

speedups

inferior

to

what

is

predicted

by

the

model

On

the

other

hand

a

message-passing

algorithm

has

explicit

accounting

of

communication

cost

and

is

more

likely

to

result

in

an

implementation

that

conforms

to

the

performance

predicted

by

the

algorithm

The

goal

of

message-passing

algorithm

is

to

enhance

locality

of

access

and

to

avoid

unnecessary

contentions

over

access

to

shared

data

The

parallel

architectures

are

generally

classified

into

two

categories

namely

Shared

address-space

architectures

-LRB-

SASAs

-RRB-

and

Message-Passing

architectures

The

SASAs

include

uniform

memory

access

-LRB-

UMA

-RRB-

architectures

-LRB-

e.g.

bus-based

symmetric

multiprocessors

-RRB-

as

well

as

distributed

memory

architectures

which

have

hardware

support

to

directly

access

remote

memory

-LRB-

e.g.

Stanford

DASH

and

Cray

-RRB-

Distributed

memory

architectures

are

also

called

non-uniform

access

-LRB-

NUMA

-RRB-

architectures

In

message-passing

architectures

access

to

a

remote

memory

location

requires

cooperation

of

the

remote

processor

and

as

a

result

is

often

orders

of

magnitude

slower

than

access

to

a

local

memory

location

Message-passing

algorithms

are

relevant

not

only

for

message-passing

architectures

but

also

for

SASAs

A

message-passing

algorithm

can

be

easily

ported

to

a

SASA

using

the

PVM/MPI

package

Unfortunately

the

performance

achieved

by

such

packages

did

not

meet

the

high-performance

requirements

of

the

GIS-range

query

problem

for

terrain

visualization

Since

each

call

to

PVM/MPI

is

treated

as

a

system

call

the

PVM/MPI

implementation

incurred

run-time

overhead

which

would

otherwise

be

avoided

To

improve

performance

we

manually

implemented

the

message-passing

algorithm

on

a

SASA

using

the

transformations

given

in

Table

The

transformation

reduced

the

overhead

incurred

in

the

PVM/MPI-based

implementation

The

transformation

converted

the

replicated

data

in

the

messagepassing

algorithm

to

shared

data

in

a

SASA

This

is

not

done

automatically

by

PVM/MPI

type

systems

This

address

space

mapping

improves

the

capacity

of

the

system

to

solve

larger

problems

relative

to

a

message-passing

architecture

-LSB-

-RSB-

Table

gives

the

mapping

of

a

message-passing

algorithm

to

a

SASA

A

message-passing

algorithm

uses

three

kinds

of

data

local

data

read-only

shared

data

and

modifiable

shared

data

The

read-only

shared

data

is

often

replicated

to

facilitate

data-sharing

particularly

if

data

transfer

is

costly

The

mapping

divides

the

shared

address-space

in

a

SASA

into

local

address

spaces

a

global

address-space

without

locks

and

a

global

address-space

with

locks

to

contain

the

three

categories

of

data

from

a

message-passing

algorithm

Message-passing

algorithms

facilitate

processor

interaction

via

point-topoint

communications

operations

such

as

Send

Receive

and

collective

communication

operations

such

as

One-to-all

broadcast

All-to-all

broadcast

and

Barrier

synchronization

These

can

be

simulated

by

shared

address

space

operations

of

read

write

lock

and

unlock

Message-Passing

algorithms

use

data

partitioning

-LRB-

e.g.

declustering

-RRB-

to

increase

locality

and

thus

reduce

processor

interaction

The

choice

of

a

declustering

method

is

largely

independent

of

the

architecture

whereas

the

choice

of

the

dynamic

load-balancing

method

is

sensitive

to

the

cost

of

processor

interaction

Table

Porting

a

message-passing

algorithm

to

SASA

Message-Passing

Algorithm

Program

for

SASAs

Address

Space

Mapping

local

data

-LRB-

read/write

-RRB-

-LRB-

non-replicated

-RRB-

local

address

space

replicated

data

-LRB-

read-only

-RRB-

global

address

space

without

locks

remote

data

-LRB-

read/write

-RRB-

global

address

space

with

locks

data-transfer/data-sharing

communication

memory

read/write

synchronization

with

messages

synchronization

with

memory

R/W

send

message

M

lock

-LRB-

M

-RRB-

write

-LRB-

M

-RRB-

unlock

-LRB-

M

-RRB-

recv

message

M

lock

-LRB-

M

-RRB-

read

-LRB-

M

-RRB-

unlock

-LRB-

M

-RRB-

one-to-all

broadcast

write

by

one

read

by

others

all-to-all

broadcast

read/write

by

all

data-partitioning

and

DLB

declustering

method

M

declustering

method

M

DLB

method

D

DLB

method

Discussion

and

Conclusions

Data-Partitioning

is

an

effective

approach

for

implementing

high

performance

in

GIS

on

SASAs

We

parallelize

the

GIS-range-query

problem

using

data-partitioning

and

dynamic

load-balancing

techniques

where

data

partitioning

is

used

to

reduce

the

synchronization

overhead

Traditional

data

partitioning

techniques

like

random

declustering

are

not

effective

for

GIS

applications

Hence

systematic

declustering

techniques

like

local

load-balancing

and

similarity-graph

method

are

needed

for

efficient

parallelization

of

GIS-range-query

problem

The

experimental

results

show

that

systematic

declustering

methods

outperform

random

declustering

methods

for

range

query

problems

Even

random

declustering

coupled

with

DLB

is

outperformed

by

SLB

methods

with

systematic

declustering

with

no

DLB

The

performance

of

DLB

methods

can

also

be

improved

by

using

the

declustering

methods

for

determining

the

subsets

of

polygons

to

be

transferred

during

run-time

In

general

a

GIS

operation

can

be

parallelized

either

on

SASAs

or

on

message-passing

architectures

But

there

are

several

advantages

to

using

SASAs

and

particularly

UMA

architectures

for

GIS

problems

when

compared

to

using

a

message-passing

architecture

Parallelizing

GIS

problems

requires

run-time

work-transfers

to

achieve

a

good

load-balance

as

the

spatial

data

objects

have

varying

sizes

and

complex

shapes

An

UMA

architecture

facilitates

these

run-time

work-transfers

as

all

the

data

is

equally

accessible

to

all

the

processors

A

disadvantage

of

the

message-passing

architecture

is

replication

of

data

across

processors

for

DLB

This

data

replication

can

also

be

avoided

in

NUMA

architectures

if

the

cost

of

remote

memory

access

is

not

much

larger

than

the

cost

of

local

access

This

data

replication

in

message-passing

architectures

reduces

the

total

main-memory

available

for

storing

the

spatial

data

References

-LSB-

-RSB-

S.

G.

Akl

and

K.

A.

Lyons

Parallel

Computational

Geometry

Prentice

Hall

Englewood

Cliffs

-LSB-

-RSB-

The

DIS

Steering

Committee

The

DIS

vision

a

map

to

the

future

of

distributed

interactive

simulation

Technical

report

Institute

for

Simulation

and

Training

May

-LSB-

-RSB-

M.

T.

Fang

R.

C.

T.

Lee

and

C.

C.

Chang

The

Idea

of

Declustering

and

its

Applications

In

Proceedings

of

the

International

Conference

on

Very

Large

Databases

-LSB-

-RSB-

J.

Goldfeather

S.

Molnar

G.

Turk

and

H.

Fuchs

Near

Real-Time

CSG

Rendering

Using

Tree

Normalization

and

Geometric

Pruning

IEEE

Computer

Graphics

and

Applications

-LRB-

-RRB-

-LSB-

-RSB-

P.

Heckbert

and

M.

Garland

Multiresolution

Modelling

for

Fast

Rendering

In

Proceedings

of

Graphics

Interface

-LSB-

-RSB-

W.

Hibbard

and

D.

Santek

Visualizing

Large

Data

Sets

in

the

Earth

Sciences

IEEE

Computer

Special

Issue

on

Vualization

in

Scientific

Computing

August

-LSB-

-RSB-

C.

Kruskal

and

A.

Weiss

Allocating

independent

subtasks

on

parallel

processors

IEEE

Transactions

on

Software

Engineering

pages

-LCB-

October

-LSB-

-RSB-

V.

Kumar

A.

Grama

A.

Gupta

and

G.

Karypis

Introduction

to

Parallel

Computing

Design

and

Analysis

of

Algorithms

The

Benjamin/Cummings

Publishing

Company

Inc.

-LSB-

-RSB-

R.

Laurini

and

D.

Thompson

Fundamentals

of

Spatial

Information

Systems

Academic

Press

Inc

-LSB-

-RSB-

D.

R.

Liu

and

S.

Shekhar

A

Similarity

Graph-Based

Approach

to

Declustering

Problem

and

its

Applications

In

Proceedings

of

the

Eleventh

International

Conference

on

Data

Engineering

IEEE

The

complete

technical

report

Csci

TR

available

via

WWW

at

URL

ps/infs.ps

-LSB-

-RSB-

D.

R.

Pratt

M.

Zyda

and

K.

Kelleher

Guest

Editor

's

Introduction

Virtual

Reality-In

the

Mind

of

the

Beholder

IEEE

Computer

Special

Issue

on

Virtual

Environments

July

-LSB-

-RSB-

S.

Shekhar

S.

Ravada

V.

Kumar

D.

Chubb

and

G.

Turner

Declustering

and

Load-Balancing

Methods

for

Parallelizing

Geographic

Information

Systems

To

appear

in

IEEE

Transaction

on

Knowledge

and

Data

Engineering

Also

available

as

Technical

Report

TR

Department

of

Computer

Science

University

of

Minnesota

URL

B I B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O B O O B I O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B I O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O B O O B I O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

declustering
geographic information systems
high performance
range query

load-balancing
shared address space architectures