Leftmost One Computation On Meshes With Row Broadcasting Himabindu Gurla Department of Computer Science , Old Dominion University , Norfolk , VA 23529 Abstract Given a matrix of size pn ? pn with every entry being a or a 1 , the leftmost one problem asks to determine the position of the leftmost 1 , if any , in each row of the matrix . The leftmost one problem finds applications in image processing , digitized geometry and computer graphics , among others . Recently , an O -LRB- n 16 -RRB- time solution to the leftmost one problem on a mesh with row buses has been proposed . However , the computational model assumes that processors have unbounded memory . The contribution of this work is to show that the problem can be solved in O -LRB- n 16 -RRB- time on a pn ? pn mesh with row broadcasting , even if each processor has only a constant number of registers . 1 Introduction The mesh-connected computer architecture has emerged as one of the natural choices for solving a large number of problems in pattern recognition , image processing , computer vision , path planning , and computational geometry -LSB- 9 -RSB- . One of the drawbacks of the mesh stems from its large computational diameter which makes the mesh architecture less attractive in contexts where computation involves data spread over processing elements far apart . A popular solution to this problem is to enhance the mesh architecture by the addition of various types of bus systems -LSB- 3 , 4 , 6 , 7 , 8 , 11 -RSB- . Early solutions involving the addition one or more global buses , shared by all the processors in the mesh , have been implemented on a number of massively parallel machines -LSB- 3 -RSB- . Recently , a more powerful architecture , referred to as mesh with row broadcasting , has been obtained by adding one bus to every row in the mesh -LSB- 6 , 10 -RSB- , see Figure 1 . The mesh with row buses has proven to be feasible to implement in VLSI , and is used in NASA 's Finite Element Machine -LSB- 3 -RSB- and in the DAP family of computers -LSB- 10 -RSB- . In recent years , efficient algorithms to solve a number of problems on meshes with broadcasting have been proposed in the literature -LSB- 1 , 2 , 4 , 5 , 6 , 12 , 13 -RSB- . Earlier , an algorithm for the leftmost one problem was proposed on meshes with row broadcasting -LSB- 13 -RSB- . The solution in -LSB- 13 -RSB- assumes that the processors have unbounded local memory . It is also claimed that the problem can be solved using processors with constant memory without giving the details and for which the extension does not appear obvious . The purpose of this note is to show that , in fact , the leftmost one problem can be solved in O -LRB- n 16 -RRB- time even if processors have a constant memory . This solution relies on an efficient list management technique which is of an independent interest . The remainder of the paper is organized as follows : section 2 discusses the list management technique used in our algorithm ; section 3 presents the details of the proposed algorithm ; finally , section 4 summarizes our findings and poses a number of open questions . 2 List Management The purpose of this section is to discuss a novel list management technique which is the key ingredient in the algorithm . Consider an array A -LSB- 1 : : N -RSB- , containing N data items , and assume that M of the data items are deleted in a sequence of M delete operations . In many contexts it is desirable to compact the list of the remaining data items . However , the compaction of an array is quite time consuming . We show that , with a judicious list management technique , a similar effect can be achieved . Although after the M delete operations the remaining data items are not compacted in A -LSB- 1 : : N ? M -RSB- , the housekeeping operations performed immediately after each delete operation ensure that the data items are shifted sufficiently far to the left in the array . In the list management technique , each delete operation , which results in one of the array elements to be set to blank , is followed by two left shift operations . The left shift operation is defined as follows . For every A -LSB- i -RSB- -LRB- 2 < = i < = M -RRB- , if A -LSB- i -RSB- is not blank and A -LSB- i ? 1 -RSB- is a blank , then A -LSB- i ? 1 -RSB- is set to A -LSB- i -RSB- , and A -LSB- i -RSB- is set to blank . We now prove that , by interleaving the delete and left shift operations , the data items remaining after the M deletes occur in the leftmost 2 -LRB- N ? M -RRB- positions in A . This is , in fact , implied by the following statement that gives more insight into the list management technique . Theorem 2.1 : For every i -LRB- 1 < = i < = M -RRB- , just prior to the ith delete operation , no prefix of A contains more blanks than data items . Proof : The proof is by induction on i. Basis step : If i = 1 , then prior to the first delete operation none of the data items are blanks , thus the claim is true for all the prefixes of A -LSB- 1 : : N -RSB- . Induction hypothesis : Assume that for any arbitrary j , -LRB- 1 < = j < = M -RRB- the statement is true just prior to the jth delete operation . It will be shown that the statement remains true just prior to the -LRB- j +1 -RRB- th delete operation . Suppose not . Let k be the smallest integer for which , just prior to the -LRB- j + 1 -RRB- th delete operation , the theorem fails . This can happen only if the following two conditions simultaneously hold : -LRB- i -RRB- The data item deleted during the jth delete operation is in A -LSB- 1 : : k -RSB- . -LRB- ii -RRB- No data item in positions k + 1 , k + 2 was shifted into the prefix A -LSB- 1 : : k -RSB- . For later reference , we make the following simple observation whose justification is immediate . Observation 2.2 : Just prior to the jth delete operation , the number of data items and blanks in the prefix A -LSB- 1 : : k -RSB- differ by at most one Since no data item enters the prefix A -LSB- 1 : : k -RSB- during the two left shifts following the jth delete operation , one of the following configurations must hold before the jth delete . Case 1 : A -LSB- k -RSB- is the last data item . Since A -LSB- 1 : : k -RSB- is a counterexample , A -LSB- k -RSB- does not shift during the two left shift operations following the jth delete operation . Note that this can happen only if both A -LSB- k ? 1 -RSB- and A -LSB- k ? 2 -RSB- are not blanks . However , by virtue of Observation 2.2 , the prefix A -LSB- 1 : : k ? 3 -RSB- would contain more blanks than data items , contradicting the induction hypothesis . Case 2 : Both A -LSB- k + 1 -RSB- and A -LSB- k + 2 -RSB- are blanks . Again by virtue of Observation 2.2 , the prefix A -LSB- 1 : : k + 2 -RSB- contains more blanks than data items , contradicting the induction hypothesis . Case 3 : A -LSB- k + 1 -RSB- is blank and A -LSB- k + 2 -RSB- contains a data item . In order to guarantee that the data item in A -LSB- k + 2 -RSB- does not shift into A -LSB- k -RSB- during the two left shift operations following the jth deletion , just prior to the jth deletion the items A -LSB- k ? 1 -RSB- and A -LSB- k -RSB- should contain data . But now the prefix A -LSB- 1 : : k ? 2 -RSB- contains more blanks than data items , which is a contradiction . Case 4 : A -LSB- k + 1 -RSB- contains a data item . Note that during the two left shift operations following the jth delete , A -LSB- k + 1 -RSB- can be stopped from entering into the prefix only if A -LSB- k ? 1 -RSB- and A -LSB- k -RSB- have data items . This again implies that just prior to the jth delete A -LSB- 1 : : k ? 2 -RSB- contains more blanks than data items , which is a contradiction . The completes the proof of the theorem For the reader 's convenience Figure 2 illustrates how the list management technique works . 3 The Algorithm This work builds on the algorithm in -LSB- 13 -RSB- that solves the leftmost one problem on meshes with row broadcasting . The solution proposed in -LSB- 13 -RSB- runs under the assumption that processors have unbounded memory and takes O -LRB- n 16 -RRB- time . Our main contribution is to show that using the list management technique discussed in Section 2 , we can perform the same task in O -LRB- n 16 -RRB- time with the processors having a constant memory . During various steps of our algorithm , we would like to view the original pn?pn mesh , R as consisting of submeshes of smaller dimensions that depend on the computational needs . We will detail them as and when the necessity arises . For further reference , note that a group of n 16 contiguous rows is termed as a slice . Specifically , the mesh R consists of n 13 slices . Step 1 : The original mesh R is partitioned into submeshes Ri ; j of size n 16 ? n 16 , each Ri ; j comprising of processors Pr ; s -LRB- 1 + -LRB- i ? 1 -RRB- n 16 < = r < = in 16 ; 1 + -LRB- j ? 1 -RRB- n 16 < = s < = jn 16 -RRB- . In each submesh Ri ; j , the position -LRB- if any -RRB- of the leftmost 1 in each row is computed and stored by the processors in the first column of Ri ; j . The processors storing this information are said to be active . If there is no 1 in a particular row , the processor in the first column of that row sets itself inactive . The information in the active processors is compacted and moved to the first row of Ri ; j along with the identity of the row from which the information comes , see Figure 3 -LRB- a -RRB- . The above operations can be performed in O -LRB- n 16 -RRB- time using local communication only . Step 2 : Now , we view the mesh in a different manner . A set of n 16 contiguous submeshes Ri ; j in a slice is termed a group . Specifically , a group Gi ; k consists of submeshes Ri ; -LRB- k ? 1 -RRB- n 16 +1 ; : : : ; Ri ; kn 16 , see Figure 3 -LRB- b -RRB- . Notice that in every Ri -LRB- 1 < = i < = n 13 -RRB- , we have exactly n 16 groups . To make the exposition clear , we will discuss the processing that takes place in one slice , say R1 . The computation is perfectly similar in all other slices . For every k -LRB- 1 < = k < = n 16 -RRB- , our aim is to dedicate the bus in row k of R1 to the group G1 ; k . In every group G1 ; k , the information in the active processors in the first row is moved to row k using local communication only . For convenience , we call a set of contiguous active processors as an active chain . Also , the leftmost processor in row k of each R1 ; j belonging to G1 ; k is called a leader . Notice that , since every group consists of n 16 Ri ; j 's , the number of leaders in any row is exactly n 16 . Step 3 : For further simplicity , we will describe the processing in row k of R1 . Since we are dedicating one bus to each group , the same processing is done in parallel in all the groups . Now , any active processor in an active chain in row k stores the identity of the row r and the position of the leftmost 1 in row r within R1 ; j it belongs to . If any active chain to the left of it declares that it has the leftmost 1 for row r , the active processor sets itself inactive . In this step , each of the n 16 leaders is responsible for broadcasting the information stored in the active chain that follows it -LRB- if any -RRB- along the bus . The leaders get their turn to broadcast sequentially from left to right . At this stage we would like to apply the list management technique described in the previous section . Specifically , a chain of active processors corresponds to the array of data items A . The broadcasts on the bus result in active processors setting themselves inactive . This corresponds to a delete operation in the chain of active processors . We start with the leftmost leader broadcasting the identity of the row r , whose informa - tion it stores . This may result in zero or more active processors setting themselves inactive including the leader itself . In each chain the creation of an inactive processor is followed by two left shift operations , just as in the list management scheme . During a left shift operation , an active processor passes on the information it contains to the inactive left neighbour -LRB- if any -RRB- and sets itself inactive . The leader continues to broadcast , as long as the chain of active processors following it is not exhausted . Each broadcast is followed by two left shift operations . Theorem 2.1 guarantees that as long as the chain of active processors is not exhausted , the leader is active and transmits the information it holds . If the leader is heading an empty list , control is passed on to the leader that is to its immediate right and so on . Note that , by virtue of Theorem 2.1 , whenever a leader gets its turn to broadcast , the length of the chain of active processors , including interspersed inactive ones caused by the deletions during the previous broadcasts , is at most twice the number of active processors in its chain . It is also clear that to determine the leftmost one in the n 16 rows , we need at most that many elements to be broadcast . From the above discussion , we can conclude that the operations in Step 3 can be accomplished in O -LRB- n 16 -RRB- time . In addition to the delete/shift following a broadcast corresponding to a row r of G1 ; k , the information is also recorded by processor Pk ; r of R1 ; 1 . Step 4 : Finally , in each Ri ; 1 -LRB- 1 < = i < = n -RRB- , in O -LRB- n 16 -RRB- time the leftmost 1 in each row can be determined using local communication . Theorem 3.1 : The leftmost one problem can be solved in O -LRB- n 16 -RRB- time on a mesh with row broadcasting , with the processors having a constant memory . Proof : The correctness of the algorithm is obvious . Since each of the Steps 1 , 2 and 4 take O -LRB- n 16 -RRB- time , and Theorem 2.1 guarantees that Step 3 can be accomplished in O -LRB- n 16 -RRB- time , we can conclude that our algorithm runs in O -LRB- n 16 -RRB- time . Also , none of the Steps require the processors to have more than a constant number of registers 4 Conclusions and open problems The significance of the leftmost one problem is due to its use in various areas including digitized geometry , image processing and computer graphics . We have shown that the leftmost one problem can be solved on a mesh with row broadcasting in O -LRB- n 16 -RRB- time , even if the processors have a constant number of registers . The lower bound argument for this problem is given in -LSB- 13 -RSB- . However , if the mesh with row broadcasting is enhanced by adding column buses , the lower bound is not known . Another interesting issue is to see whether the list management technique can be applied to problems like finding the maximum of items in each row of the mesh . References -LSB- 1 -RSB- D. Bhagavathi , P. J. Looges , S. Olariu , J. L. Schwing , and J. Zhang , A fast selection algorithm on meshes with multiple broadcasting , Proc . International Conference on Parallel Processing , 1992 , St-Charles , Illinois , III-10-17 . -LSB- 2 -RSB- D. Bhagavathi , S. Olariu , W. Shen , and L. Wilson , A Time-Optimal Multiple Search Algorithm on Enhanced Meshes , with Applications , Proc . Fourth Canadian Computational Geometry Conference , St-Johns , August 1992 , 359-364 . -LSB- 3 -RSB- S. H. Bokhari , Finding maximum on an array processor with a global bus , IEEE Trans . on Computers vol . C-33 , no. 2 , Feb. 1984 . 133-139 . -LSB- 4 -RSB- Y. C. Chen , W. T. Chen , G. H. Chen and J. P. Shen , Designing efficient parallel algorithms on mesh connected computers with multiple broadcasting , IEEE Trans . Parallel and Distributed Systems , vol . 1 , no. 2 , Apr. 1990 . -LSB- 5 -RSB- V. P. Kumar and C. S. Raghavendra , Array processor with multiple broadcasting , Journal of Parallel and Distributed Computing , vol 2 , 1987 , 173-190 . -LSB- 6 -RSB- V. P. Kumar and D. I. Reisis , Image computations on meshes with multiple broadcast , IEEE Trans . on Pattern Analysis and Machine Intelligence , vol . 11 , no. 11 , 1989 , 1194 - 1201 . -LSB- 7 -RSB- H. Li and M. Maresca , Polymorphic-torus network , IEEE Transactions on Computers , vol . C-38 , no. 9 , -LRB- 1989 -RRB- 1345 -LCB- 1351 . -LSB- 8 -RSB- M. Maresca and H. Li , Connection autonomy and SIMD computers : a VLSI implementation , Journal of Parallel and Distributed Computing , vol . 7 , 1989 , 302-320 . -LSB- 9 -RSB- D. Nassimi and S. Sahni , Finding Connected Components and Connected Ones on a Mesh-Connected Parallel Computer , SIAM Journal on Computing , 9 -LRB- 1980 -RRB- , 744-757 . -LSB- 10 -RSB- D. Parkinson , D. J. Hunt , and K. S. MacQueen , The AMT DAP 500 , 33rd IEEE Comp . Soc . International Conf. , Feb. 1988 , 196-199 . -LSB- 11 -RSB- J. Rothstein , Bus automata , brains , and mental models , IEEE Trans . on Systems Man Cybernetics 18 , 1988 . -LSB- 12 -RSB- Q.F.Stout , \ Mesh connected computers with broadcasting , '' IEEE Transactions on Computers , vol . C-32 , Sept. 1983 , pp. 826-830 . -LSB- 13 -RSB- | | , \ Meshes with multiple buses , '' in Proc . 27th IEEE Symp . Found . Comput . Sci. , 1986 , pp. 264-273 . Figure 1 : A mesh with row broadcasting of size 4 ? 5 1 3 4 5 9 10 11 13 15 1 3 4 5 9 10 11 13 15 1 3 4 5 9 10 11 13 15 after 1st left shift 4 1 3 4 5 9 10 11 13 15 1 3 4 5 9 10 11 13 15 1 3 4 5 9 10 11 13 15 1 3 4 5 9 10 11 13 15 1 3 4 5 9 10 11 13 15 after 2nd left shift 4 5 after 3 is deleted after 5 is deleted 4 after 1st left shift 4 after 2nd left shift 9 10 after 1 is deleted 4 4 9 10 4 9 10 11 after 1st left shift 11 94 after 2nd left shift 11 10 94 after 13 is deleted 11 94 after 1st left shift 9 10 94 10 11 after 2nd left shift 10 15 15 15 A -LSB- 1 . .9 -RSB- 9 13 15 13 15 Figure 2 : An Illustration of the list management technique for a sequence of M = 4 delete operations on A -LSB- 1 : :9 -RSB- , each delete followed by two two left shift operations . Note that just prior to any delete operation , the number of blanks in any prefix is no more than the number of data items . Rij Rij R1 R1 ... ? Represents active processors ? Represents inactive processors ? Represents leaders of active chains of processors G1 G2 G3 G n 1/6 R11 R12 -LRB- a -RRB- -LRB- b -RRB- R2 . . . . . . . Rn 1/3 R 2n1/6 +1 R n1/6 +22 R 3n1/6 ... . R 1n1/3 1 , 1,1 , Figure 3 : Illustrating the various data movements in the Algorithm

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

computer graphics
digitized geometry
image processing
meshes with row broadcasting
parallel algorithms


RMSP : A Reliable Multicast Session Protocol for Collaborative Continuous-Feed Applications Walid Mostafa Mukesh Singhal Department of Computer and Information Science The Ohio State University Columbus , OH 43210 email : -LRB- mostafa , singhal -RRB- @cis . ohio-state . edu Phone : 614-292-5839 Fax : 614-292-2991 Abstract This report presents a portable reliable multicast session protocol for collaborative continuousfeed applications . The protocol supports NxN group multicast communications at the session layer and runs on the top of existing reliable transport protocols -LRB- e.g. , TCP -RRB- . It does not assume multicast support at the network layer . The protocol is portable as a network-layerindependent library that is suitable for both LAN and WAN environments . It multiplexes messages from different group members into transport packets to reduce the transport overhead per message . It requires only two transport-level connections per node , irrespective of the multicast group size , to reduce the overhead of acknowledgements and retransmissions per node . We compare the throughput of this protocol with the throughput of general sender-initiated and receiver-initiated reliable multicast transport protocols . 1 Introduction The continuous growth in applications that require communications among a group of nodes or processes has led to considerable interest in multicast communication . These applications span distributed computing environments -LRB- e.g. , ISIS -LSB- Bir93 -RSB- and Transis -LSB- ADKM92 -RSB- -RRB- , audio/video multicast -LRB- e.g. , MBONE -LSB- Eri94 -RSB- -RRB- , and distributed whiteboards -LRB- -LSB- McC92 -RSB- -RRB- . The quality of service required by multicast applications vary . While some applications are sensitive to response time but may tolerate data loss -LRB- e.g. , multimedia -LSB- SCFJ96 -RSB- and video conferencing -LSB- Jac94 -RSB- -RRB- , other types of applications require reliable receipt of all data even if it means longer response time -LRB- e.g. , software distribution and file transfers -RRB- . In this report , we consider a class of applications that we call continuous-feed collaborative applications . This is an NxN communication paradigm where each node continuously generates and/or collects data and is required to send it reliably in-order to all other nodes in the multicast group . In other words , each node must have a full representation of the data generated/collected by all other nodes . These applications require reliable transport of data and may tolerate longer response times . The NxN connection throughput is a crucial performance index since each node must support the traffic rates generated by all other nodes in the multicast group . Such applications may include regional satellite-feed stations , stock market centers , intelligent network control points sharing service subscribers updates , telecommunication switches multicasting changes to load and routing information , mobile centers tracking subscribers , sites with replicated or mirrored files , and others . These applications may run on public , enterprise , or private networks with different network layers . Our goal is to provide a high-throughput protocol for reliable NxN multicast communications . As network bandwidth is increasing at a faster pace than processing speed -LSB- PTK94 -RSB- , protocol processing overhead at a node is expected to be a bottleneck . Therefore , a protocol must reduce the processing overhead at a node . Node processing overhead is influenced by what protocol layer multicasting and reliable delivery of messages are provided . Our approach to provide high-throughput NxN connections is different from existing protocols . We isolate the multicast communication management issues from the reliable transport issues . We use a session layer to manage multicast communication and use an existing reliable point-to-point transport layer to provide reliable delivery . We reduce the transport-level processing overhead in two ways : a -RRB- each node requires only two transportlevel connections to reduce the overhead of acknowledgements and retransmissions per node , and b -RRB- we multiplex messages from different nodes into transport-level packets to reduce the overhead per message . We introduce a reliable multicast session protocol , RMSP . The following are the highlights of the RMSP protocol that distinguish it from the existing protocols : ffl directly supports NxN group communications ffl manages multicast communications at the session level ffl multiplexes several session-level messages per transport-level packet ffl requires only two transport-level connections per node ffl does not assume multicast support at the network level ffl distributes the protocol processing overhead uniformly among all nodes ffl A sender receives one positive acknowledgement per message on the session level This report presents an overview and an analytic model for a throughput analysis of the RMSP protocol . We compute the mean processing time per multicast message at a sender and at a receiver . We also compute the maximum throughput of an NxN multicast connection . We then compare the NxN connection throughput of the RMSP protocol with those of the sender-initiated and receiver-initiated reliable multicast transport protocols of -LSB- PTK94 -RSB- . Section 2 describes how existing protocols address multicast communications . Section 3 describes the RMSP protocol . Section 4 presents a throughput analysis of RMSP . Section 5 provides a performance comparison between the throughput of the RMSP protocol and those of -LSB- PTK94 -RSB- . Summary and concluding remarks are presented in Section 6 . 2 Background Existing reliable multicast transport protocols , e.g. , -LSB- PSK94 -RSB- , address the issue of reliable delivery of packets at the transport level . They assume that the network layer provides an unreliable multicast datagram delivery service -LRB- e.g. , IP multicast -LSB- Dee89 -RSB- -RRB- for efficient use of the network bandwidth . Since the network layer is connectionless , these protocols have to maintain all multicast information , e.g. , group membership and delivery trees , at the transport level , thus adding to its complexity . Such protocols provide NxN multicast connections by using N independent 1x -LRB- N ? 1 -RRB- multicast transport connections . Reliable delivery of data is concerned with packet loss detection and retransmission . It can be the responsibility of the sender or the receivers -LRB- -LSB- PTK94 -RSB- -RRB- . Sender-initiated protocols require a sender to receive positive acknowledgements , ACKS , from all receivers per packet . Receiver-initiated protocols rely on each receiver to detect and request missing packets using negative acknowledgements , NAKs . There are several variations to these two schemes to improve their efficiency . In one variation , the multicast group is structured in a tree-like hierarchy where designated nodes in the subtrees are responsible for consolidating acknowledgements and processing local retransmission -LRB- e.g. , -LSB- YGS95 -RSB- -RRB- . Another variation use restricted multicasting to localize the scope of NAKs and retransmissions -LRB- e.g. , -LSB- FJM +95 -RSB- -RRB- . A third variation uses log servers to process retransmission requests -LRB- -LSB- HSC95 -RSB- -RRB- . In 1xN communication models , one sender multicasts messages to N receivers . The 1xN connection throughput is limited by the minimum of the sender throughput and the receiver throughput . Pingali et al. . compared one sender-initiated , A , and two receiverinitiated , N1 and N2 , reliable multicast transport protocols in -LSB- PTK94 -RSB- . They showed that the sender processing time is higher than the receiver processing time for all three protocols and , therefore , the throughput of their 1xN connections are bound by the throughput of their senders . In order to improve the 1xN connection throughput , existing protocols attempt to improve the throughput of a sender by reducing its processing time per message . Such improvement may result in an increase in the processing time at a receiver . Pingali et al. -LSB- PTK94 -RSB- showed that protocol N2 yields higher throughput than protocol N1 . This is because the throughput of a sender using the N2 protocol is higher than that of a sender using the N1 protocol -LRB- even though the reverse is true for receivers -RRB- . NxN communication models are different . In these models , each node multicasts messages to -LRB- N ? 1 -RRB- receivers and receives multicast messages from -LRB- N ? 1 -RRB- senders . If all nodes are multicasting at the same rate , a node receives -LRB- N ? 1 -RRB- multicast messages for every message it sends . Therefore , the maximum throughput supported by a node is -LRB- N ? 1 -RRB- times more sensitive to receiver processing time than to sender processing time . Existing protocols provide an NxN connection by using N independent 1x -LRB- N ? 1 -RRB- connections -LRB- e.g. , -LSB- LP96 -RSB- -RRB- . Using this model , the NxN connection throughput of protocol N1 of -LSB- PTK94 -RSB- is higher than that of protocol N2 although the latter has higher 1x -LRB- N ? 1 -RRB- connection throughput . 3 Protocol Description RMSP goals are portability , scalability , and efficiency . Portability is realized by providing RMSP as a network-layer-independent session-layer library that supports both LAN and WAN environments . Scalability is achieved through configuring the multicast group into a virtual pipelined chain where each member requires only two transport-level connections irrespective of the group size . Efficiency accrues from multiplexing independent session-level messages into transport packets to reduce the transport overhead per message . RMSP arranges the multicast group nodes into a virtual ring where each node has a previous and next neighbors . Each node establishes a transport-level connection with each neighbor . A node sends transport packets in one direction to its next neighbor . Figure 1 shows a multicast group with four nodes -LRB- A ; B ; C ; D -RRB- . The transport paths A ! B ; B ! C ; C ! D ; and D ! A constitute the NxN connection or virtual ring . RMSP maintains local node session C D BA TCP Application IP RMSP Network B C DA TCP Application IP RMSP Network A D B CA TCP Application IP RMSP Network D B A C D TCP Application IP RMSP Network B C remote node session Figure 1 : RMSP Multiplexed Session Layer session id session message seq # Message session message length session message data Transport Packet Data transport sender id transport receiver id Transport Packet Headertransport packet length other transport info : : : # of multiplexed session messages session id session message seq # session message length session message data Session Message Header Session Message Data Session Figure 2 : Transport Packet with Multiplexed Session Messages all the multicast communication information at the session level . The transport layer has no knowledge of the multicast group . This is the main factor that contributes to RMSP portability . A session is a sequence of messages that are generated by a node 's application layer . RMSP has a data structure for every session in the multicast group to maintain its state information -LRB- e.g. , last message received , last message sent , last message acknowledged , etc. -RRB- . These data structures are represented by the gray and black nodes in Figure 1 . RMSP multiplexes several session messages in a transport packet to reduce the transport layer overhead per message . Figure 2 illustrates the structure of a transport packet . Its header is a typical point-to-point transport header that describes the source , the destination , and other packet information . Since RMSP runs on the top of the existing reliable transport layer , it does not modify any of its parameters . However , the data payload of the transport packet is managed by RMSP . The first field indicates the number of session messages multiplexed within this transport packet and is followed by the session messages . Each session message has a header identifying which session it belongs to , the message 's relative sequence number within that session , and the length of that message . RMSP uses an outgoing buffer , B , to prepare data for the next outgoing transport-level packet . The buffer size represents the multiplexing factor , i.e. , the maximum number of session messages allowed in one transport-level packet . RMSP maintains a timer to control how long it can wait before it sends the outgoing buffer to the transport layer . The timer is set when RMSP puts a session message into an empty buffer . RMSP sends the outgoing buffer to the transport layer when the buffer is full or when the timer expires . Both the buffer size and the timer value are defined at connection setup . The buffer has a timer associated with it . The timer specifies the maximum time that RMSP can wait before sending the buffer to the transport layer . When RMSP receives a transport-level packet from the transport layer , it demultiplexes the session messages . If a message originated from this node , RMSP updates the data structure for this node to reflect the acknowledgement of the message then discards it . Otherwise , RMSP updates the data structure for the sender to reflect the receipt of the message , forwards a copy to the application , and puts the message in the outgoing buffer , B . When RMSP receives a message to multicast from the application layer , it prepares a session header for it and puts it in the outgoing buffer , B. C ,5 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 A B C A - > B B - > C A < - C A , A , A , A , A , A , A , A , A , A , B , B , B , B , B , B , B , B , B , B , C , C , C , C , C , C , C , C , C , C , A ,1 C ,1 A ,2 B ,1 C ,2 A ,3 B ,2 C ,3 A ,4 A ,5 C ,4 B ,3 B ,1 A ,1 B ,2 B ,3 A ,2 C ,1 C ,2 A ,3 B ,4 B ,5 A ,4 C ,3 C ,1 B ,1 C ,2 B ,2 A ,1 C ,3 C ,4 B ,3 A ,2 A ,3 B ,4 Figure 3 : RMSP Transport Packet Flow with Multiplexed Session Messages Figure 3 shows a multicast example for a group of 3 nodes using RMSP . Node A multicasts session message -LRB- A ; 1 -RRB- by sending it in a transport packet to node B. Node B demultiplexes message -LRB- A ; 1 -RRB- , sends it to its application layer , and puts it in an outgoing transport buffer . Node B also has message -LRB- B ; 1 -RRB- ready , so it puts it in the outgoing transport buffer as well . Node B sends the outgoing buffer to the transport layer that , in turn , sends the transport packet to node C . When message -LRB- A ; 1 -RRB- returns to its sender , node A , it treats as an acknowledgement and removes it from the multicast pipe . Connection Management RMSP assumes a connection manager that multicast group members will register with before a connection is setup . The connection manager arranges the members in a virtual ring . It informs each member with both previous and next ring neighbors and sends each member a list of the multicast group members as well . Figure 1 shows a multicast group with four nodes -LRB- A ; B ; C ; D -RRB- . Each node establishes a transport-level connection with its previous and next neighbors . If a new member wishes to join the multicast group , it sends a request to the connection manager . The manager determines the new member 's location on the virtual ring and informs its previous and next neighbors to break their transport-level connection and establish new ones with the new member . The manager sends the new member 's information to the multicast group . A similar scenario , but in reverse order , takes place when a member leaves the group . RMSP achieves flow control by using a rate-based mechanism on the session level similar to the one used in -LSB- YGS95 -RSB- . This prohibits applications from sending session messages to RMSP faster than some predefined maximum transmission rate . The transport level flow control is provided by the transport layer -LRB- e.g. , window-based mechanism in TCP -RRB- . RMSP does not modify the transport layer flow control mechanism . Reliability RMSP runs on the top of an existing reliable transport layer that guarantees reliable and sequenced delivery of transport packets . RMSP provides a positive acknowledgement on the session level . This is achieved when a session message travels along the virtual ring and returns back to the sender . RMSP can use mechanisms such as heartbeat -LRB- -LSB- HSC95 -RSB- -RRB- for node failure detection . Each node periodically reports its status to the connection manager . If the manager detects that a node is not responding , it instructs the two neighbors around it to break their transport-level connections with the faulty node and establish a connection between themselves . When a node fails , transport packets in progress may be lost . When a node fails , a transport packet is lost if the node returned an acknowledgement to its previous neighbor but did not forward it successfully to its next neighbor . Message loss detection is achieved by using the session acknowledgement . When a message , say i , returns to the sender , it treats it as an acknowledgement and expects message -LRB- i + 1 -RRB- to return next . If message -LRB- i + 2 -RRB- returns to the sender , the RMSP layer recognizes that message -LRB- i + 1 -RRB- is lost and retransmits it . 4 RMSP Throughput Analysis In this section , we present an analytic modeling of the RMSP protocol and derive expressions for the maximum throughput and the mean processing time of session messages at a node . We include the processing time and throughput results derived by Pingali et al. . -LSB- PTK94 -RSB- for the generic sender-initiated and receiver-initiated reliable multicast transport protocols . We use notations similar to those used in -LSB- PTK94 -RSB- . 4.1 Multicast Group Throughput We assume that the multicast group consists of N identical nodes . Each node generates traffic at rate of > = session messages . Therefore , each node needs to multicast session messages successfully at a rate > = , and receive multicast session messages from other nodes at rate > = -LRB- N ? 1 -RRB- . Hence , if a node uses a processing time of X and Y to successfully send and receive one multicast session message , respectively , then E -LSB- Z -RSB- , where E -LSB- Z -RSB- = > = E -LSB- X -RSB- + > = -LRB- N ? 1 -RRB- E -LSB- Y -RSB- -LRB- 1 -RRB- is a measure of the mean node utilization and it must satisfy -LRB- E -LSB- Z -RSB- < = 1 -RRB- for the node to be able to meet the throughput demand . Therefore , the maximum throughput per session per node , > = , must satisfy the following condition > = < = 1 = -LRB- E -LSB- X -RSB- + -LRB- N ? 1 -RRB- E -LSB- Y -RSB- -RRB- : -LRB- 2 -RRB- 4.2 RMSP Throughput The multicast communication model we use is NxN where the multicast group has N nodes , all the nodes are identical , and the application layer on each node generates session messages at rate > = . The RMSP protocol arranges the nodes into a virtual ring . Every node gets involved in receiving , demultiplexing , multiplexing , and sending a session message . p * this overhead for the transport message -LRB- F session messages -RRB- Ack Start X f Ys X no s * Xp * Yp * X t a Y * Xa a Y * * Yp Ys Y f p * Xp * X t * Xa * Originating Node Next Node -LRB- sender -RRB- -LRB- receiver -RRB- Previous -LRB- receiver -RRB- Node timeout timeout yes yes -LRB- 1-p -RRB- no -LRB- 1-p -RRB- Figure 4 : RMSP Sender Node Processing Time To determine the maximum throughput that a node can support -LRB- equation 2 -RRB- , we compute the mean processing time required to successfully send and receive a multicast session message . We consider one node sending one session message to the other -LRB- N ? 1 -RRB- nodes . We also consider that node as a receiver of a multicast session message sent by another node . Finally we consider that node sending multicast session messages at rate > = , and receiving multicast messages at rate > = from each other node . Figure 4 illustrates a session message flow through transport packets from the sender node to the next node , across the pipe to the previous node , and then back to the sender node . In order to multicast a session message from a sender , the application layer passes the message to the RMSP session layer with processing time Xf . The RMSP layer multiplexes the message , along with other sessions messages , into the current transport packet with processing time Xs and passes the packet to the transport layer . The transport layer , in turn , transmits the packet to the next node with processing time Xp . If the packet is lost , with probability p , the transport layer will timeout then retransmit the packet using processing times of Xt and Xp , respectively . We define M such that E -LSB- M -RSB- represents the expected number of packet transmissions from one node until it is successfully received by the other node . The session message eventually returns to the sender in another transport packet from previous node where the transport layer acknowledges it and forwards it to the RMSP session layer with processing times Ya and Yp , respectively . The RMSP session layer demultiplexes the message , updates this node 's data structure , and removes the message from the multicast pipe with processing time Ys . We emphasize that any processing time associated with the transport packet should be divided by the multiplexing factor , F , when attributed to a session message . This applies to Xp ; Xt ; Xa ; Yp ; Yt ; and Ya . Hence , a sender 's processing time to successfully multicast a session message may be expressed as XR = Xf + Xs + 1 F Xp + 1 F M X m = 2 -LRB- Xt + Xp -RRB- + 1 F Xa + 1 F -LRB- Yp + Ya -RRB- + Ys : Since we are interested in the mean processing time to transmit one session message reliably between the sender node and the next node , E -LSB- XR -RSB- is expressed as E -LSB- XR -RSB- = E -LSB- Xf -RSB- + E -LSB- Xs -RSB- + E -LSB- Ys -RSB- + 1 F -LRB- E -LSB- M -RSB- E -LSB- Xp -RSB- + -LRB- E -LSB- M -RSB- ? 1 -RRB- E -LSB- Xt -RSB- + E -LSB- Xa -RSB- + E -LSB- Yp -RSB- + E -LSB- Ya -RSB- -RRB- : -LRB- 3 -RRB- The random variable M is binomially-distributed with success probability -LRB- 1 ? p -RRB- . There - fore , the expected value of M , E -LSB- M -RSB- , is given as follows : E -LSB- M -RSB- = 1 X m = 1 mP -LSB- M = m -RSB- = 1 X m = 1 mpm ? 1 -LRB- 1 ? p -RRB- = 1 -LRB- 1 ? p -RRB- -LRB- 4 -RRB- substituting -LRB- 4 -RRB- in -LRB- 3 -RRB- we get E -LSB- XR -RSB- = E -LSB- Xf -RSB- + E -LSB- Xs -RSB- + E -LSB- Ys -RSB- + 1 F -LRB- 1 ? p -RRB- -LRB- E -LSB- Xp -RSB- + E -LSB- Xt -RSB- -RRB- + 1 F -LRB- E -LSB- Yp -RSB- + E -LSB- Ya -RSB- + E -LSB- Xa -RSB- ? E -LSB- Xt -RSB- -RRB- : -LRB- 5 -RRB- For a RMSP receiver , e.g. , next node in Figure 4 , the processing time to receive and acknowledge a transport packet carrying a designated session message is Yp and Ya , respectively . The RMSP session processing time to demultiplex the session message is Ys and to forward it upward to the application is Yf . The receiver forwards the transport packet to the next node and process the corresponding acknowledgement with processing time Yp and Ya , respectively . Yp and Ya are transport-level overhead and should be divided by the multiplexing factor F when attributed to a session message . Thus , a receiver 's processing time to successfully receive and process a session message can be expressed as Y R = Yf + Ys + 1 F -LRB- Yp + Ya -RRB- + 1 F Xp + 1 F M X m = 2 -LRB- Xt + Xp -RRB- + 1 F Xa where M is an identical random variable to the sender case -LRB- i.e. , binomial distribution with an expected value of 1 = -LRB- 1 ? p -RRB- as in Equation -LRB- 4 -RRB- -RRB- . Therefore , the expected value of Y R is E -LSB- Y R -RSB- = E -LSB- Yf -RSB- + E -LSB- Ys -RSB- + 1 F -LRB- 1 ? p -RRB- -LRB- E -LSB- Xp -RSB- + E -LSB- Xt -RSB- -RRB- + 1 F -LRB- E -LSB- Yp -RSB- + E -LSB- Ya -RSB- + E -LSB- Xa -RSB- ? E -LSB- Xt -RSB- -RRB- : -LRB- 6 -RRB- The maximum throughput per node , in terms of number of successfully sent/received session messages , is the reciprocal of the processing time for one message . Therefore , the maximum throughput by a sender node and a receiver node are 1 = E -LSB- XR -RSB- and 1 = E -LSB- Y R -RSB- , respectively . The maximum throughput per session per node for RMSP , > = R , can now be expressed using Equation -LRB- 2 -RRB- as > = R = 1 = -LRB- E -LSB- XR -RSB- + -LRB- N ? 1 -RRB- E -LSB- Y R -RSB- -RRB- -LRB- 7 -RRB- where E -LSB- XR -RSB- and E -LSB- Y R -RSB- were derived in Equations -LRB- 5 -RRB- and -LRB- 6 -RRB- . It is important to observe that the size of the multicast group , N , does not appear in any of the processing times for either sender or receiver nodes -LRB- that 's Equations -LRB- 5 -RRB- and -LRB- 6 -RRB- , respectively -RRB- . This result is a fundamental to the scalability of the RMSP protocol . In other words , processing time per session message at any node is independent of the multicast group size . To compare a sender and a receiver processing time per session message , we express E -LSB- XR -RSB- in terms of E -LSB- Y R -RSB- from Equations -LRB- 5 -RRB- and -LRB- 6 -RRB- as follows : E -LSB- XR -RSB- = E -LSB- Y R -RSB- + E -LSB- Xs -RSB- + E -LSB- Xf -RSB- ? E -LSB- Yf -RSB- : -LRB- 8 -RRB- It is interesting to observe that in Equation -LRB- 8 -RRB- , the difference between the sender processing time , E -LSB- XR -RSB- , and the receiver processing time , E -LSB- Y R -RSB- , for a multicast session message is a slight overhead of E -LSB- Xs -RSB- assuming that E -LSB- Xf -RSB- and E -LSB- Yf -RSB- are comparable . This is an important RMSP result that shows how symmetric and balanced the sender and receiver processing times are for a multicast session message . This asserts that the processing load over all multicast nodes is comparable irrespective of the multicast traffic load presented by each node . 4.3 RMSP Variations RMSP may be implemented in different ways depending on the objective . In this section , we present two variations of RMSP . Both variations forbid a sender from receiving its session message back to serve as a logical acknowledgement . The first variation has better throughput while the second variation improves connection response time . In the first variation of RMSP , a session message is removed from the multicast pipe at the node just before the original sender . Since that node 's RMSP will have to de-multiplex the session message and identify its sender before passing it to the application layer , it will delete it instead of multiplexing it in the outgoing transport packet . RMSP does that only if it recognizes that the sender of the message is its next node neighbor . Consequently , the processing times of the sender and the receiver are reduced and the node 's throughput improves . The mean processing time for the sender in this case is E -LSB- XR1 -RSB- = E -LSB- Xf -RSB- + E -LSB- Xs -RSB- + 1 F -LRB- E -LSB- Xa -RSB- ? E -LSB- Xt -RSB- -RRB- + 1 F -LRB- 1 ? p -RRB- -LRB- E -LSB- Xp -RSB- + E -LSB- Xt -RSB- -RRB- and the mean processing time for the receiver is E -LSB- Y R1 -RSB- = E -LSB- Yf -RSB- + E -LSB- Ys -RSB- + 1 F -LRB- E -LSB- Yp -RSB- + E -LSB- Ya -RSB- -RRB- + -LRB- N ? 1 -RRB- NF -LRB- E -LSB- Xa -RSB- ? E -LSB- Xt -RSB- + E -LSB- Xp -RSB- + E -LSB- Xt -RSB- -LRB- 1 ? p -RRB- -RRB- : Therefore , the maximum throughput per session per node , > = R1 , for this variation is > = R1 = 1 = -LRB- E -LSB- XR1 -RSB- + -LRB- N ? 1 -RRB- E -LSB- Y R1 -RSB- -RRB- -LRB- 9 -RRB- In another variation , a sender multicasts a session message in both directions of the virtual ring instead of one direction only . In RMSP , transport traffic flows in one direction while acknowledgement traffic flows in the other . In this variation , transport traffic flows in both directions , and therefore , acknowledgements can be piggybacked on it . A receiver still forwards a transport packet only to the next node along the direction of the travel of the packet . A session message traveling along both directions of the sender eventually reaches two neighboring receivers that will forward it to each other . In such cases , these two receivers will remove that message from the transport pipe after recognizing that it is a duplicate . Nodes receive session messages in shorter time than in the original RMSP . However both sender and receiver processing times will be slightly higher per message and consequently , the throughput will be slightly lower . The mean processing time for the sender is E -LSB- XR2 -RSB- = E -LSB- Xf -RSB- + E -LSB- Xs -RSB- + 2 F -LRB- E -LSB- Xa -RSB- ? E -LSB- Xt -RSB- -RRB- + 2 F -LRB- 1 ? p -RRB- -LRB- E -LSB- Xp -RSB- + E -LSB- Xt -RSB- -RRB- and the mean processing time for the receiver is E -LSB- Y R2 -RSB- = E -LSB- Yf -RSB- + 1 F -LRB- E -LSB- Xa -RSB- ? E -LSB- Xt -RSB- -RRB- + 1 F -LRB- 1 ? p -RRB- -LRB- E -LSB- Xp -RSB- + E -LSB- Xt -RSB- -RRB- + -LRB- 1 + 2 -LRB- N ? 1 -RRB- -RRB- -LRB- E -LSB- Ys -RSB- + 1 F -LRB- E -LSB- Yp -RSB- + E -LSB- Ya -RSB- -RRB- -RRB- : Therefore , the maximum throughput per session per node , > = R2 , for this variation is > = R2 = 1 = -LRB- E -LSB- XR2 -RSB- + -LRB- N ? 1 -RRB- E -LSB- Y R2 -RSB- -RRB- -LRB- 10 -RRB- 4.4 Generic RMTP Throughput Pingali et al. compared the throughput of one sender-initiated and two receiver-initiated reliable multicast transport protocols , RMTPs , in -LSB- PTK94 -RSB- . These protocols support 1xN multicast connections . The 1xN connection throughput is the minimum of the sender throughput and the receiver throughput . We use their results to derive the NxN connection throughput for their protocols using Equation -LRB- 2 -RRB- . In the sender-initiated protocol , a sender maintains an ACK list for each packet it multicasts . The per-packet ACK list has a status field per receiver that gets updated when a positive ACK is received from that receiver . The sender re-multicasts the packet if any receiver does not respond within a certain time . A receiver returns an ACK for every packet it receives even if it is a duplicate . The mean processing time for a sender to successfully multicast a packet , E -LSB- XA -RSB- , is given next . E -LSB- XA -RSB- = E -LSB- M -RSB- E -LSB- Xp -RSB- + -LRB- E -LSB- M -RSB- ? 1 -RRB- E -LSB- Xt -RSB- + RE -LSB- M -RSB- -LRB- 1 ? p -RRB- E -LSB- Xa -RSB- + E -LSB- Xf -RSB- : -LRB- 11 -RRB- The mean processing time for a receiver to successfully receive that packet , E -LSB- Y A -RSB- , is given by E -LSB- Y A -RSB- = E -LSB- M -RSB- -LRB- 1 ? p -RRB- E -LSB- Yp -RSB- + E -LSB- M -RSB- -LRB- 1 ? p -RRB- E -LSB- Ya -RSB- + E -LSB- Yf -RSB- -LRB- 12 -RRB- where E -LSB- M -RSB- = RX i = 1 R i ! -LRB- ? 1 -RRB- i +1 1 -LRB- 1 ? pi -RRB- : -LRB- 13 -RRB- Using Equation -LRB- 2 -RRB- , the maximum throughput of session messages per node may be expressed as > = A = 1 = -LRB- E -LSB- XA -RSB- + -LRB- N ? 1 -RRB- E -LSB- Y A -RSB- -RRB- : -LRB- 14 -RRB- In the receiver-initiated protocol , a sender multicasts packets continuously until it receives a NAK for a packet from a receiver , in which case it re-multicasts the packet again . The receiver uses a timer to detect a lost packet and sends a NAK to the sender over a point-topoint connection . The mean processing time for a sender to successfully multicast a packet , E -LSB- XN -RSB- , is given next . E -LSB- XN -RSB- = E -LSB- Xf -RSB- + E -LSB- M -RSB- E -LSB- Xp -RSB- + pR -LRB- 1 ? p -RRB- E -LSB- Xn -RSB- : -LRB- 15 -RRB- The mean processing time for a receiver to successfully receive that packet , E -LSB- Y N -RSB- , is given by E -LSB- Y N -RSB- = E -LSB- Yf -RSB- + E -LSB- M -RSB- -LRB- 1 ? p -RRB- E -LSB- Yp -RSB- + p -LRB- 1 ? p -RRB- E -LSB- Yn -RSB- + p2 -LRB- 1 ? p -RRB- E -LSB- Yt -RSB- -LRB- 16 -RRB- where E -LSB- M -RSB- is expressed in -LRB- 13 -RRB- . The maximum throughput of session messages per node is > = N = 1 = -LRB- E -LSB- XN -RSB- + -LRB- N ? 1 -RRB- E -LSB- Y N -RSB- -RRB- : -LRB- 17 -RRB- 5 A Performance Comparison We now compare the throughput of RMSP with the protocols in -LSB- PTK94 -RSB- . We are interested in the sensitivity of these protocols to multicast parameters such as the group size , N , the transport packet loss probability , p , and the multiplexing factor , F . We use processing time assumptions and parameter values similar to -LSB- PTK94 -RSB- . We set all processing times to 1 . Thus , E -LSB- Xi -RSB- = E -LSB- Yj -RSB- = 1 , where -LRB- i ; j -RRB- 2 ff ; s ; p ; t ; a ; ng . We assume that processing one transport packet packet with F session messages in RMSP is comparable to processing one transport packet in RMTP -LRB- the longer bytes to copy is offset by simpler connection management , 2 instead of N , and much fewer context switches -RRB- . Figure 5 shows how the sender processing time per message is impacted by the group size and the multiplexing factor when p is :1 . RMSP processing time is constant and does not depend on N . This is because a sender transmits a message only to the next node and receives it back from the previous node . The RMSP sender processing time decreases as the multiplexing factor increases until it asymptotically reaches a constant value . This is because the sender processing time is O -LRB- 1 + 1F -RRB- . The sender processing time for other protocols increase as the group size increases . This is because a sender has to process more retransmission requests . The sender processing time for the ACK protocol is higher than that of the NAK protocol because of the processing of positive acknowledgements . They are both an order of magnitude or more higher than the RMSP sender processing time . Figure 6 shows how the receiver processing time per message is impacted by the group size and the multiplexing factor when p is :1 . RMSP processing time is constant and does not depend on N because a receiver receives a message only from the previous node and forwards it to the next node . The RMSP receiver processing time decreases as F increases until it asymptotically reaches a constant value . The receiver processing time for the ACK protocol is higher than that of the NAK protocol because it sends a positive acknowledgement for every packet it receives even if it is a duplicate . The RMSP receiver spends more processing time per message than the NAK protocol receiver when F is 1 for any group size . This is the case of no multiplexing where the transport-level overhead per message is not reduced . The RMSP receiver processing time is less than that of the NAK protocol when F is 4 or higher . Figures 5 and 6 show that the processing time of the sender is higher than that of the receiver for both the ACK and the NAK protocols . These two protocols support a 1xN communication model . The 1xN connection throughput is the minimum of the sender throughput and the receiver throughput . Therefore , a sender is the 1xN connection bottleneck for both protocols . Pingali et al. -LSB- PTK94 -RSB- showed that the NAK protocol , referred to as N1 in their paper , has much higher throughput than the ACK protocol . They also showed that the throughput of another receiver-initiated protocol they presented , N2 , is higher than that of N1 . However , we are interested in the NxN connection throughput as expressed by Equation 2 . This equation shows that the weight of the receiver processing time is -LRB- N ? 1 -RRB- times the weight of the sender processing time . Using Equation 2 , N1 yields higher throughput than N2 for an NxN connection -LRB- although the former yields higher throughput for 1xN connections -RRB- . This is an important result because it shows that techniques used to improve the throughput of 1xN communications may not necessarily improve NxN communications . Thus , we use N1 , or NAK , for comparison with RMSP . Figure 7 shows how the sender processing time is impacted by the packet loss probability and the multiplexing factor when N is 500 . RMSP sender processing time is O -LRB- 1 + 1 F -LRB- 1 ? p -RRB- -RRB- and is relatively insensitive to p. For low values of p , a sender of the ACK protocol processes positive acknowledgements . As p increases , senders of both the ACK and the NAK protocols process more retransmissions and the difference between their processing times decreases . Figure 8 shows how the receiver processing time is impacted by the packet loss and the multiplexing factor when N is 500 . As p increases , the receiver of the ACK protocol receives more duplicate packets and sends more positive acknowledgements . The receiver of the NAK protocol sends more negative acknowledgements and receives more duplicate packets . The RMSP receiver processing time is O -LRB- 1 + 1 F -LRB- 1 ? p -RRB- -RRB- and is relatively insensitive to p as well . Figure 9 shows how the ratio of the throughput the RMSP protocol to the throughput of the NAK protocols varies with the group size and the multiplexing factor when p is :1 . RMSP throughput is higher than that of NAK when F is 4 for any group size . Figure 10 shows how the throughput ratio of the RMSP and the ACK protocols varies with the packet loss and the multiplexing factor when N is 500 . It also shows the ratio of the throughput of the NAK and the ACK protocols . Note that the throughput of the RMSP protocol is higher than that of the ACK protocol except when F is 1 and p < :005 . The throughput ratio between RMSP and ACK steadily increases as p increases . This is because the ACK protocol accrues more timeout and retransmission overhead while RMSP is not sensitive to p . The throughput of the NAK protocol is higher than that of the ACK protocol for all loss probabilities . This is because the processing times of a sender and a receiver of the NAK protocol are lower per message than those of the ACK protocol . Figure 11 shows how the throughput ratio of the RMSP and the NAK protocols varies with the multiplexing factor when N 2 f100 ; 1000g and p 2 f0 :01 ; :1 g . The increase in the throughput ratio is high for lower values of F and slows down to asymptotically reach a constant value for high values of F . This is because the processing time of a sender and a receiver is O -LRB- 1 + 1F -RRB- . Figure 12 shows that the RMSP variation , that eliminates a session message at the node before the sender , yields higher throughput than RMSP for all cases . This is because it eliminates the transport overhead at the node just before the sender and eliminates the transport and session overheads at the sender 's node . This throughput improvement attains the highest when F is 1 . This is because the transport overhead per session message is the highest . As F increases , the throughput ratio decreases until it aymptotically reaches a constant value . 6 Summary and Concluding Remarks RMSP is a reliable multicast session protocol that supports NxN multicast communications . RMSP is suitable for collaborative continuous-feed applications that require reliable data delivery and can tolerate longer response time . It is portable on LAN and WAN environments as a network-layer-independent session-layer library and does not require multicast support at the network level . It reduces the transport layer overhead by requiring only two transportlevel connections per node and by multiplexing several session-level messages per transportlevel packet . It distributes the protocol processing overhead uniformly among all nodes to achieve high throughput . In this report , we developed an analytic model to compute the mean throughput of RMSP for NxN multicast connections . We analyzed its sensitivity to group size , transport packet loss probability , and session multiplexing factor . We compared RMSP throughput to the throughput of the sender-initiated and the receiver-initiated reliable multicast transport protocols presented in -LSB- PTK94 -RSB- . We showed that RMSP is less sensitive to group size and packet loss than these protocols . We also showed that RMSP yields higher throughput than the other protocols for a wide range of multicast parameters -LRB- i.e. , multicast group size , packet loss , and multiplexing factor -RRB- . 1 10 100 1000 10000 200 400 600 800 1000 Sender Processing Time/Multicast Message Multicast Group Size p = 0.1 RMSP F = 64 RMSP F = 8 RMSP F = 2 RMSP F = 1 NAK ACK Figure 5 : Sender processing time vs. group size 1 2 3 4 5 6 7 8 9 200 400 600 800 1000 Receiver Processing Time/Multicast Message Multicast Group Size p = 0.1 RMSP F = 64 RMSP F = 8 RMSP F = 2 RMSP F = 1 NAK ACK Figure 6 : Receiver processing time vs.group size 22 1 10 100 1000 10000 0.001 0.01 0.1 1 Sender Processing Time/Multicast Message Transport Packet Loss Probability N = 500 RMSP F = 64 RMSP F = 8 RMSP F = 2 RMSP F = 1 NAK ACK Figure 7 : Sender processing time vs. loss probability 2 4 6 8 10 12 0.001 0.01 0.1 1 Receiver Processing Time/Multicast Message Transport Packet Loss Probability N = 500 RMSP F = 64 RMSP F = 8 RMSP F = 2 RMSP F = 1 NAK ACK Figure 8 : Receiver processing time vs. loss probability 23 0.5 1 1.5 2 2.5 200 400 600 800 1000 Throughput Ratio -LRB- RMSP/NAK -RRB- Multicast Group Size p = 0.1 F = 128 F = 64 F = 8 F = 4 F = 2 F = 1 Figure 9 : Throughput ratio vs. group size 1 2 3 4 5 6 7 8 0.001 0.01 0.1 1 Throughput Ratio -LRB- NAK/ACK & RMSP/ACK -RRB- Transport Packet Loss Probability N = 500 RMSP/ACK F = 64 RMSP/ACK F = 8 RMSP/ACK F = 1 NAK/ACK Figure 10 : Throughput ratio vs. loss probability 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 1 10 100 Throughput Ratio -LRB- RMSP/NAK -RRB- Multiplexing Factor -LRB- F -RRB- N in -LCB- 100,1000 -RCB- & p in -LCB- 0.01,0.1 -RCB- 1000 , 0.1 100 , 0.1 1000,0.01 100,0.01 Figure 11 : Throughput ratio vs. multiplexing factor 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 1 10 100 Throughput Ratio Multiplexing Factor -LRB- F -RRB- p = 0.1 RMSP/NAK N = 500 Modified RMSP/NAK N = 500 Modified RMSP/RMSP Figure 12 : Modified RMSP throughput ratio vs. multiplexing factor References -LSB- ADKM92 -RSB- Y. Amir , D. Dolev , S. Kramer , and D. Malki . \ Transis : A Communication Sub-System for High Availability '' . In 22nd Annual International Symposium on Fault-Tolerant Computing -LRB- FTCS -RRB- , July 1992 . -LSB- Bir93 -RSB- K. Birman . The process group approach to reliable distributed computing . Communications of the ACM , 36 -LRB- 12 -RRB- :37 -LCB- 53 , December 1993 . -LSB- Dee89 -RSB- S. Deering . Host extensions for ip multicasting . Internet Request for Comment -LRB- RFC 1112 -RRB- , August 1989 . -LSB- Eri94 -RSB- H. Eriksson . Mbone : The multicast backbone . Communications of the ACM , 37 -LRB- 8 -RRB- :54 -LCB- 60 , August 1994 . -LSB- FJM +95 -RSB- S. Floyd , V. Jacobson , S. McCanne , C. Liu , and L. Zhang . \ A Reliable Multicast Framework for Light-weight Sessions and Application Level Framing '' . In Proceedings of ACM SIGCOMM ' 95 . ACM , August 1995 . -LSB- HSC95 -RSB- H. Holbrook , S. Singhal , and D. Cheriton . \ Log-Based Receiver-Reliable Multicast for Distributed Interactive Simulation '' . In ACM SIGCOMM Conference . ACM , August 1995 . -LSB- Jac94 -RSB- Van Jacobson . Multimedia conferencing on the internet . In SICGOMM 1994 Tutorial Notes . ACM , August 1994 . -LSB- LP96 -RSB- J. Lin and S. Paul . \ RMTP : A Reliable Multicast Transport Protocol '' . In INFOCOM ` 96 Conference . IEEE , March 1996 . -LSB- McC92 -RSB- S. McCanne . A distributed whiteboard for network conferencing . DSG Working Paper , 1992 . -LSB- PSK94 -RSB- S. Paul , K. Sabnani , and D. Kristol . Multicast transport protocols for high speed networks . In Proceedings of International Conference on Network Protocols , October 1994 . -LSB- PTK94 -RSB- S. Pingali , D. Towsley , and J.F. Kurose . \ A Comparison of Sender-Initiated and Receiver-Initiated Reliable Multicast Protocols '' . In SIGMETRICS '94 Conference . ACM , May 1994 . -LSB- SCFJ96 -RSB- H. Schulzrinne , S. Casner , R. Frederick , and V. Jacobson . Rtp : A transport protocol for real-time applications . Internet Request for Comment -LRB- RFC 1889 -RRB- , January 1996 . -LSB- YGS95 -RSB- R. Yavatkar , J. Griffioen , and M. Sudan . \ A Reliable Dissemination Protocol for Interactive Collaborative Applications '' . In ACM Multimedia Conference . ACM , 1995 .

O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

reliable multicast
session multiplexing
transport protocols