APPARC Deliverable PME4b Performance Analysis of Multiprocessors Memory System ? O. Temam University of Versailles 45 Av . des Etats-Unis 78000 Versailles France Abstract The purpose of this paper is to study the feasibility of the performance evaluation and prediction of numerical codes in a multiprocessor environment . This work is divided into two complementary parts . First an experimental section for determining the average communication time of a data within a numerical loop nest for different types of communications . The purpose of this experimental section is also to quantify the notion of false sharing and moreover of data layout . In a second part , a formal framework is developed for estimating the amount of communication time for a given loop nest because of the initial data layout of the data . This framework relies on the experimental data gathered in the first part . It is shown that estimating the total communication time for a set of loop nests is a tractable task and has numerous applications -LRB- parallelization , software coherence : : : -RRB- 1 Introduction The purpose of this work is to study the feasibility of cache performance analysis and prediction in the context of multiprocessors . This work has already been conducted for uniprocessors -LSB- 1 -RSB- , but it is not straightforward that it can be extended to multiprocessors . Besides , there were many studies on the nature , importance of each type of cache misses for uniprocessors -LSB- 2 , 3 -RSB- , so that the difficulties in elaborating a cache model -LRB- i.e. , cache interferences -RRB- could be easily identified . It is not the case for multiprocessors , where few detailed analysis of the cache behavior have been conducted . The purpose of performance analysis is to provide an understanding of the behavior of a computer under usual workloads . Since it is difficult and confusing to detail the performance of each subset of the computer system , performance analysis should be restricted to unveiling the assets and flaws of a computer system . This can be phrased differently as giving a feeling of the behavior of the computer system . Our purpose here is to analyze the memory system of a multiprocessor under a numerical workload . Such codes are characterized by the use of arrays which induce large-scale data movements between processors . We are particularly concerned with shared-memory multiprocessors or distributed-memory multiprocessors with a virtual shared memory which are particularly promising because of their ease ? This work was supported by the Esprit Agency DG XIII under Grant No . APPARC 6634 BRA III . of use , but which are limited by coherence and communications management . Consequently , there is a strong need for better understanding and evaluating communication issues in such computers . On a first step , a set of synthetic loops is provided that can be used to evaluate the performance of a given memory hierarchy for each type of communications 2 . These results provide intuitions on the influence of communications on the multiprocessor system performance . By pointing out the most time-consuming and frequent constructs , these results can be used to focus an analytical approach to performance analysis and prediction , as shown in section 3 2 Experimental Analysis of a Multiprocessor Memory System 2.1 Experimental Framework The experiments of this paper have been conducted on the KSR multiprocessor . The KSR is an allcache distributed memory machine with a hardware-implemented shared virtual memory . Each node contains a superscalar processor and a two-level cache hierarchy . The first-level data cache -LRB- subcache -RRB- is 256-kbyte large , while the second-level data cache is 32-Mbyte large . The coherence is enforced at the level of a subpage which is 128-byte large . One of the assets of the KSR , with respect to performance evaluation , is to provide hardware monitors that indicate the number of communications per thread , the communication time and several other statistics on the memory system . The KSR compiler provides a number of parallelization directives that allow the use different types of scheduling : block scheduling , cyclic scheduling , block cyclic scheduling , wavefront : : : In this paper , block scheduling is used . 2.1.1 Main Types of Communications With respect to performance evaluation , loop nests can be divided into the following categories : Loop nest with a dependence . ffl The dependence requires that the loop nest be serialized -LRB- 1 -RRB- ffl The loop nest can be parallelized but suffers from true sharing -LRB- 2 -RRB- . ffl The loop nest can be split in independent sets after being restructured -LRB- 3 -RRB- . Loop nest without dependences . -LRB- 4 -RRB- Case -LRB- 1 -RRB- sums up to a monoprocessor problem , and the reader is referred to -LSB- 1 -RSB- , where the performance evaluation of a monoprocessor cache memory is conducted . If a loop nest corresponds to case -LRB- 2 -RRB- , it clearly constitutes a performance bottleneck . The purpose of this paper is to analyze less wellunderstood cases that fall into cases -LRB- 3 -RRB- and -LRB- 4 -RRB- . For cases -LRB- 3 -RRB- and -LRB- 4 -RRB- , the communications that can occur are divided into the following categories : Coherence : Coherence costs occur when a processor needs to modify a data that it does not own . In this case , it must first invalidate the copies of other processors . The test program should only measure the time required to invalidate the data , not to fetch it . Consider the following program : C * sequential do 1 j = 1 , n a -LRB- j -RRB- = 1.0 1 continue C * parallel do 2 j = 1 , n b -LRB- j -RRB- = a -LRB- j -RRB- 2 continue C * parallel do 3 j = 1 , n a -LRB- j -RRB- = a -LRB- j -RRB- + 1.0 3 continue C * parallel do 4 j = 1 , n a -LRB- j -RRB- = a -LRB- j -RRB- + 1.0 4 continue The purpose of loop 1 is to lay out the entire array a in the cache of the master processor . Loop 2 distributes a read-only copy of slices of array a to the slaves processor . In loop 3 , each processor must request the ownership of the slice of array a of which it already has a copy . Loop 4 is identical to loop 3 , except that each processor already owns its copy of the slice of a. Therefore the ratio Tcoherence = T3T4 provides a measure of the average time necessary to maintain coherence per data element . This ratio is independent of n -LRB- this has been observed experimentally ; it is a reasonable hypothesis assuming a linear increase of T3 and T4 with n -RRB- . On the other hand , it is a function of P , the number of processors . Indeed , the number of processors sending coherence requests simultaneously influence the average time to service such a request by modifying the network contention . Also , the average distance between two processors -LRB- a parameter of distributed-memory architectures -RRB- is dependent on P . Figure 2.1.1 shows the influence of P on Tcoherence . Communications due to coherence are significant . Therefore , evaluating the memory system performance can not sum up to counting communications indifferently , it is necessary to distinguish between read and write requests . Note that it is important to measure the cost of maintaining coherence when P processors instead of 1 are active , because the parallelization of numerical loops often split a loop nest in P symmetric parts , so that the P processors are likely to perform similar actions simultaneously and induce a high network traffic . Communications Independently of the ownership of a data , a fundamental parameter is the time required to obtain a data from a distant processor . The parallelization of numerical loop nests can induce several typical cases . Either P processors request data which are all located in a single processor memory , or a single processor might request data located in the memories of P processors . The test programs below illustrate these two cases . 1 ! P Consider the following program : C * sequential do 1 j = 1 , n a -LRB- j -RRB- = 1.0 1 continue C * parallel 1 1.2 1.4 1.6 1.8 2 2.2 2.4 1 5 10 15 20 25 30 32 Execution time -LRB- coherence -RRB- / -LRB- no coherence -RRB- # of procs 3 3 3333333 3 3333 3 3 3 3 3 333 3 33333333 Figure 1 : Influence of coherence on communication time . do 2 j = 1 , n b -LRB- j -RRB- = a -LRB- j -RRB- 2 continue C * parallel do 3 j = 1 , n b -LRB- j -RRB- = a -LRB- j -RRB- 3 continue The reasonning is similar to the previous paragraph . Loop 1 lays array a in the memory of the master processor , while in loop 2 each processor requests a read-only copy of a slice of array a. Loop 3 is identical to loop 2 except that each processor already has a copy of its slice of a. Therefore Tcom -LRB- 1 , P -RRB- = T2T3 provides a measure of the average time required to get a data from a distant processor . Again , this parameter is a function of P which determines the contention of the network and the distance between processors . P ! 1 Consider the following program : C * parallel do 1 j = 1 , n a -LRB- j -RRB- = 1.0 1 continue 4 C * sequential do 2 j = 1 , n b -LRB- j -RRB- = a -LRB- j -RRB- 2 continue C * sequential do 3 j = 1 , n b -LRB- j -RRB- = a -LRB- j -RRB- 3 continue Tcom -LRB- P ,1 -RRB- = T2 P?T3 provides the average time required to bring the data located in the P processors to the master processor . This test program isolates the influence of the distance between processors , since the time to fetch a data can be dependent on the processor it is located on . As a consequence , when P increases Tcom -LRB- P ,1 -RRB- might increase depending on the architecture . Figure 2.1.1 shows the influence of P on Tcom -LRB- P ,1 -RRB- , and figure 2.1.1 shows the influence of P on Tcom -LRB- 1 , P -RRB- . It appears that the cost of communications 1 ! P increases much faster than P ! 1 which increases quasi-linearly . On the other hand , communications P ! 1 reach a higher maximum than communications P ! 1 . In the first case -LRB- 1 ! P -RRB- , the cost increases quickly with the number of processors because several processors request data simultaneously , inducing high network traffic . In the second case -LRB- P ! 1 -RRB- , since the loop is sequential , no contention occurs , but on the other hand the average distance between the master processor and the processor having the data increases regularly with the number of processors . This problem is felt less strongly in the first case because requests are simultaneous . 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1 5 10 15 20 25 30 32 Execution time -LRB- 1 ! P -RRB- / -LRB- no communication -RRB- # of procs 3 3 3 3 333333 33333 3 3 3 333333 3333 33 3 Figure 2 : Communications 1 ! P . 1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 1 5 10 15 20 25 30 32 Execution time -LRB- P ! 1 -RRB- / -LRB- no communication -RRB- # of procs 3 3 3333 3 3 33 33 33 33333 3 3 33 3 3 3 3 33333 Figure 3 : Communications P ! 1 . False Sharing When two processors modify the same page but use different data on this page -LRB- e.g. different parts of the same array -RRB- , the page is alternately requested by each processor inducing a ping-pong phenomenon which is called false sharing . False sharing can be a major cause of performance degradation in numerical codes , so that its influence must be carefully evaluated . Consider the following program : C * sequential do 1 j = 1 , P * Ps a -LRB- j -RRB- = 1.0 1 continue C * parallel do 2 j = 1 , P do 2 k = 1 , n a -LRB- S * j -RRB- = a -LRB- S * j -RRB- + 1.0 2 continue C * parallel do 3 j = 1 , P do 3 k = 1 , n a -LRB- Ps * j -RRB- = a -LRB- Ps * j -RRB- + 1.0 3 continue 6 In loop 1 , array a is located in the master processor memory . Due to the numerous data movements induced by false sharing , the initial placement of data in this case has little influence . Consider loop 2 and assume that S = 1 . Then each page of a must be modified simultaneously by P processors -LRB- or more accurately by min -LRB- P ; Ps -RRB- processors where Ps is the page size expressed in words -RRB- , i.e. , false sharing occurs . The purpose of loop k is to ensure that each processor uses a page of a long enough that false sharing can effectively occur . The importance of false sharing is determined by the number of processors P which are victim of false sharing , but also by the number of processors accessing simultaneously the same page . These two parameters are different . The first parameter determines the number of processors using the network because of false sharing , while the second parameter actually determines the intensity of false sharing for each page . Parameter S , i.e. , the step , is used to vary the intensity of false sharing . Each page of a is accessed simultaneously by min -LRB- Ps = S ; P -RRB- processors . Therefore , in loop 3 where S = Ps , no false sharing occurs , and Tfalse = T2T3 provides a measure of the cost of false sharing . Tfalse is a function of both P and S as explained above . In figure 4 , it appears that the performance degradation due to false sharing increases in a quasi-linear manner with the number of processors , when false sharing is intense , i.e. , when S is low . When S is small , the performance degradation increases in a sub-linear manner . Now for a fixed number of processors , the intensity of false sharing decreases hyperbolically with parameter S . This behavior is predictable since min -LRB- Ps = S ; P -RRB- processors access a page simultaneously . In absolute value , the performance degradation of false sharing is very important -LRB- the execution time is up to 35 times higher -RRB- , but the case shown in the above loops is the worst possible , and it can be avoided with simple compile-time analysis . Several loop nests extracted from real numerical benchmarks were run on the KSR , and false sharing proved to have little impact on the loop nests performance -LRB- an example can be found in section 2.2 -RRB- . The main reason is that the KSR subpage is fairly small -LRB- 128 bytes -RRB- ; false sharing would have a much stronger impact if the page size is large . 5 10 15 20 10 20 30 40 5 10 15 20 25 30 35 S # of procs Execution time -LRB- S -RRB- / -LRB- S = 16 -RRB- Figure 4 : Influence of False Sharing . Data Layout The second main issue of multiprocessor communications is data layout . In a program , each loop nest is parallelized separately , so that different portions of the same array can be used by a processor from one loop nest to another . As a consequence , a processor often has to fetch a data on another processor the first time it is referenced . Consider the following program : C * sequential do 1 j = 1 , n a -LRB- j -RRB- = 1.0 1 continue C * parallel do 2 j = 1 , n do 2 k = 1 , m a -LRB- j -RRB- = a -LRB- j -RRB- + 1.0 2 continue C * parallel do 3 j = 1 , n do 3 k = 1 , m a -LRB- j -RRB- = a -LRB- j -RRB- + 1.0 3 continue The purpose of loop 1 is again to lay array a on the master processor . In loop 2 , each processor fetches a slice of array a . If m = 1 , the ratio Tlayout = T2T3 measures the average time required to fetch and own a data . The purpose of parameter m is to determine how many reuses of the data are necessary before the impact of the initial distribution can be hidden . Tlayout is shown as a function of m in figure 5 for different values of P . Figure 5 shows that the influence of the data layout decreases very rapidly with the reuse of the element . Basically , after m = 10 reuses , the influence of the data layout does not feel . However , when m = 1 , the influence of the data layout is strong -LRB- the execution time is multiplied by 2 -RRB- . Actually , many arrays in numerical loop nests are not reused so that the case m = 1 occurs much more frequently than m large . Consequently , though data layout influence is not as intense as for false sharing , its frequency of occurrence is such that it should be considered with the same care . 2.2 An Example of Real Numerical loop nest The purpose of this section is to illustrate the influence of communications on a real numerical loop nest . This loop nest is extracted from the routine EFLUX of FLO52 a Perfect Club benchmark -LSB- 4 -RSB- . C * parallel DO 3 N = 1,4 DO 3 J = 2 , JL DO 3 I = 2 , IL DW -LRB- I , J , N -RRB- = FS -LRB- I , J , N -RRB- - FS -LRB- I-1 , J , N -RRB- 3 CONTINUE 5 10 15 20 25 30 20 15 10 5 1 1.5 2 # procs m Execution time -LRB- m -RRB- / -LRB- m = 1 -RRB- Figure 5 : Influence of Data-Layout . C * parallel DO 5 J = 2 , jl DO 5 I = 2 , IL XX = X -LRB- I , J ,1 -RRB- - X -LRB- I-1 , J ,1 -RRB- YX = X -LRB- I , J ,2 -RRB- - X -LRB- I-1 , J ,2 -RRB- PA = P -LRB- I , J +1 -RRB- + P -LRB- I , J -RRB- QSP = -LRB- XX * W -LRB- I , J +1,3 -RRB- - YX * W -LRB- I , J +1,2 -RRB- -RRB- / W -LRB- I , J +1,1 -RRB- QSM = -LRB- XX * W -LRB- I , J ,3 -RRB- - YX * W -LRB- I , J ,2 -RRB- -RRB- / W -LRB- I , J ,1 -RRB- FS -LRB- I , J ,1 -RRB- = QSP * W -LRB- I , J +1,1 -RRB- + QSM * W -LRB- I , J ,1 -RRB- FS -LRB- I , J ,2 -RRB- = QSP * W -LRB- I , J +1,2 -RRB- + QSM * W -LRB- I , J ,2 -RRB- - YX * PA FS -LRB- I , J ,3 -RRB- = QSP * W -LRB- I , J +1,3 -RRB- + QSM * W -LRB- I , J ,3 -RRB- + XX * PA FS -LRB- I , J ,4 -RRB- = QSP * -LRB- W -LRB- I , J +1,4 -RRB- + P -LRB- I , J +1 -RRB- -RRB- + QSM * -LRB- W -LRB- I , J ,4 -RRB- . + P -LRB- I , J -RRB- -RRB- 5 CONTINUE In loop nest 3 the code is parallelized according to variable N , while in loop nest 5 , it is parallelized accordint to variable J . As a consequence , in loop nest 3 array FS is distributed orthogonally to the way it is used in loop nest 5 . This results in numerous communications for array FS . By repeating twice loop nest 5 , and measuring the time difference between the communication time spent in the first and the second execution of this loop nest , it is possible to isolate the time spent because of the initial data distribution -LRB- assuming , that the data layout is correct for the second execution -RRB- . Therefore , for the first execution of loop nest 5 , the communication time corresponds to data layout and false-sharing , while for the second execution it corresponds to false-sharing only . The two corresponding graphs can be seen in figure 2.2 . As can be seen , data layout has a stronger effect on the total communication time than false sharing . When the coherence is enforced at a small granularity , as in the KSR system , this 9 case probably often occurs . 1 2 3 4 5 6 7 8 9 10 11 1 5 10 15 20 25 30 32 Communication time -LRB- in 10 ? 4 seconds -RRB- # of procs False-Sharing and Data-Layout 3 3 33 3 333 33 3 333 3 3 33 3 3 3 3 3 33 33 3 33 3 3 3 False-Sharing + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Figure 6 : Communication time for a real numerical loop nest . 2.3 Observations on the Performance Analysis of a Numerical Loop Nest Estimating the stall time due to the memory system for a numerical loop nest can be decomposed into two problems : a uniprocessor and a multiprocessor problem . The uniprocessor problem consists in evaluating the number of capacity , conflict and compulsory misses . Capacity and conflict misses occur during execution of the loop nest and do not require inter-processor communications , except in the case of false-sharing -LRB- which is not addressed in this paper ; see the next sections -RRB- . Compulsory misses correspond to the first reference to data . If the data is already present in the processor local memory , then the whole problem of estimating the stall time due to the memory system is a uniprocessor problem . If some of the data is not present in the local memory than it is a multiprocessor problem with respect to the subset of non-local data . The issue is to determine the number of data that must be fetched and the average time to fetch such data . In this experimental section , we have determined the average time to fetch a data as a function of the number of processors . We have also shown that the initial data layout has a strong impact on a loop nest performance . The remaining issue , which is addressed in the next section , is to determine the total communication time required to fetch the data , using these experimental parameters . 3 A Formal Approach to the Evaluation of Data Communications Since only numerical codes are considered , the problem of evaluting the communication time is restricted to numerical loop nests . However , in opposition to the majority of existing research works , we will not be concerned with intra-loop but with inter-loop communications . As explained in section 2 , intra-loop communications can either be formulated as an uniprocessor problem , which was addressed in -LSB- 1 -RSB- , or a false sharing problem which is addressed in -LSB- 5 -RSB- . Addressing the inter-loop communications problem requires the development of a new framework for modeling this type of communications . This framework is presented in section 3.1 . In section 3.2 , a method is proposed for estimating the inter - loop communication time between two consecutive loop nests , based on this framework . In section 3.3 , applications of this framework are discussed . 3.1 Characterizing Array Elements within a Processor Memory Let us consider the following loop nest . DO j1 = 1 , N DO j2 = 1 , N ... = T -LRB- j1 , j2 -RRB- ENDDO ENDDO The iteration space associated with this loop nest is defined by the following set of inequations 1 < = j1 < = N 1 < = j2 < = N More generally , the loop bounds define a set of constraints that characterize the iteration space accessed by a loop . These constraints can be represented by a system of inequations such as A ~ | < = b Geometrically , this set of inequations defines a bounded convex polyhedron P P = f ~ | = A ~ | < = bg : However , there is not a one-to-one mapping between the integer points laying within P and the portion of array T used by a reference T -LRB- f -LRB- j1 ; : : : ; jn -RRB- -RRB- if ker -LRB- f -RRB- < > ; -LSB- 6 -RSB- , i.e. , if the reference exhibits temporal reuse . The set of elements of T used by this reference is defined by the image of P by the affine function f -LRB- j1 ; : : : ; jn -RRB- -RRB- . As mentioned in -LSB- 7 -RSB- , the image of a polyhedron by an affine function is a polyhedron . So , the problem is to determine the new set of constraints defining the polyhedron Q = f -LRB- P -RRB- . This study is restricted to array subscripts of the form -LRB- ji1 + 1 ; : : : ; jir + r -RRB- where ik is an integer constant , and jik is one of fj1 ; : : : ; jng and 8ik ; ik0 ; k < > k0 .1 Therefore , if F is the matrix associated with the affine function f , r = rg -LRB- F -RRB- , and 9U ; V , n ? n matrices / F = U Ir ! V . 3.1.1 Image of a polyhedron by an affine function In this section , we want to determine the matrix C and the vector d such that Cy < = d , y 2 f -LRB- P -RRB- , where P = fx = Ax < = bg . Intuitively , the purpose is to find the image of the iteration space polyhedron by the affine function defining an array subscript . Characterizing y = f -LRB- x -RRB- Let f = fl + f0 where fl is the linear function associated with f , and f0 an affine point . Given a vector y 2 Im -LRB- fl -RRB- , we want to find a vector x , such that y = fl -LRB- x -RRB- . Note that , if F is singular , there exist multiple such vectors x. Proposition 3.1 Let y 2 Im -LRB- fl -RRB- a vector of dimension r , then 9U 1 an r ? n matrix , 9x01 a vector of dimension r such that x01 = U 1y : 1As shown in -LSB- 8 -RSB- , this type of indices encompass about 90 % of the subscripts found in real numerical codes . Proof If y + f0 2 f -LRB- P -RRB- then y 2 Im -LRB- fl -RRB- . Let y0 = U ? 1y and x0 = V x , then y0 = y01 y02 ! = Irx01 ! , i.e. , y01 = x01 , where x01 and y01 are vectors of dimension r . If we note U ? 1 = U 1 U 2 ! , where U 1 is an r ? n matrix , then y01 = U 1y , so that x01 = U 1y . 2 Since y is independent of x02 , it is possible to choose x02 = 0 . Intuitively , the U ; V matrices transform the canonical basis into a basis f1 ; : : : ; fn where f1 ; : : : ; fr is a basis of Im -LRB- fl -RRB- and fr +1 ; : : : ; fn is a basis of ker -LRB- fl -RRB- . Characterizing x 2 P Proposition 3.2 Let P a polyhedron defined by fx ; Ax < = bg , then 9A01 an m ? r matrix , 9U 1 an r ? n matrix such that y + f0 2 Q = f -LRB- P -RRB- , A01U 1 -LRB- y + f0 -RRB- < = b : Proof It has been seen that x 2 P , Ax < = b , where A is an m?n matrix . Ax < = b , AV ? 1x0 < = b -LRB- see above paragraph -RRB- . Let AV ? 1 = -LRB- A01A02 -RRB- . Then , since x02 = as previously mentioned , AV ? 1x0 = A01x01 . And Ax < = b , A01x01 < = b. Therefore , it is now possible to state that if y + f0 2 f -LRB- P -RRB- , then A01U 1 -LRB- y + f0 -RRB- < = b . The sufficient condition can be proved easily . Consequently , the polyhedron Q = f -LRB- P -RRB- is defined as follows y + f0 2 Q = f -LRB- P -RRB- , A01U 1 -LRB- y + f0 -RRB- < = b : 2 Note that the matrix A01F 0 ? 1U 1 of the system is an m ? r matrix , i.e. , the number of constraints m is the same as in the original matrix m , but only r variables need be considered . Since matrices U ; V ; V ? 1 ; F ; F 0 ? 1 can be found by application of the Gaussian elimination method , the problem of finding the image Q = f -LRB- P -RRB- of P can be solved in polynomial time . 3.1.2 Polyhedra and parallel loops A slightly more complex task is to provide a formal characterization of the data residing in the local memory -LRB- or cache -RRB- of a multiprocessor node when a loop is parallelized . Consider the tiled version of the previous loop . There are P processors , indexed by the variable p with < = p < = P ? 1 . DO jj1 = 1 , N , N/P DOALL j1 = jj1 , min -LRB- jj1 + N/P -1 , N -RRB- DO j2 = 1 , N ... = T -LRB- j1 , j2 -RRB- ENDDO ENDDO By extending the polyhedron framework , it is possible to characterize the portion of the iteration space distributed to each processor using parameter p. For instance , in the previous loop , the polyhedron Pp associated with processor p is the following . -LRB- p ? 1 -RRB- ? NP < = j1 < = p ? NP 1 < = j2 < = N In this example the direction of parallelization is -LRB- 1,0,0 , : : : -RRB- . Let us consider the general case where a direction of parallelization ~ d is used . As mentioned in previous paragraphs , the polyhedron associated with the loop nest iteration space is characterized by a system of inequations A ~ | < = b. Parallelizing the loop according to a given direction is equivalent to splitting the iteration space using regularly spaced affine hyperplanes -LRB- see figure 7 -RRB- . The equation of such hyperplanes is t ~ dx = c where c is a constant , and ~ d is the direction orthogonal to the hyperplanes . The different hyperplanes are parallel and only differ by parameter c , c 2 -LSB- ld ; ud -RSB- . The bounds ld ; ud are defined by ld = min ? t ~ d ~ j ; A ~ j < = b ? ud = max ? t ~ d ~ j ; A ~ j < = b ? Direction d Iteration space polyhedron Figure 7 : Parallelizing according to direction ~ d. So , for processor p ; -LRB- 1 < = p < = P -RRB- , the associated portion of the iteration space is defined by -LRB- p ? 1 -RRB- ? ud ? ld P + ld < = ~ d : ~ | < = p ? ud ? ld P + ld : These two inequations can be added to the previous system along with variable p , so that the polyhedron Pp distributed to each processor can be described with the new system A ~ | < = bp , where A = B @ A t ~ d ? t ~ d 1 CA , and bp = B @ b -LRB- p ? 1 -RRB- ? ud?ld P + ld p ? ud?ld P + ld 1 CA . 3.1.3 Array space accessed by a processor Therefore , the set of elements of an array T accessed by a reference T -LRB- f -LRB- j1 ; : : : ; jn -RRB- -RRB- within the memory of processor p can be characterized with the polyhedron Qp = f -LRB- Pp -RRB- . The system of inequations associated with Qp is computed by the method of section 3.1 . Consequently , the number of array elements accessed by this processor is equal to the number of integer points S -LRB- Qp -RRB- within the rational polyhedron Qp . This number can be computed with techniques similar to those presented in -LSB- 9 -RSB- . However , the next sections show that knowing the exact number of points is not necessary , so that computing an approximation of this number is all that is needed . A reasonable approximation can be given by S -LRB- Qp -RRB- ' vol -LRB- Qp -RRB- , i.e. , the volume of the rational polyhedron . The precision of this approximation is discussed in section 3.2.4 and in -LSB- 9 -RSB- . 3.2 Estimating the Amount and the Cost of Communications In this section the problem of evluating the number of communications between two consecutive loop nests is addressed . In the first section , an example is used to illustrate the technique used . In the second section , a general formulation of this technique is presented . Further issues are addressed in the last section . 3.2.1 Number of inter-loop communications Let us consider the following program c Nest 1 DOALL j1 = 1 , N DO j2 = 1 , N ... = T -LRB- f -LRB- j1 , j2 -RRB- -RRB- ENDDO ENDDO c Nest 2 DOALL j1 = 1 , N DO j2 = 1 , N ... = T -LRB- g -LRB- j1 , j2 -RRB- -RRB- ENDDO ENDDO All processor memories are assumed to be empty . In nest 1 , elements of T are fetched by each processor . The number of elements fetched is given by S -LRB- Qfp -RRB- , where Qfp is the polyhedron characterizing the elements of array T used by reference T -LRB- f -LRB- : : :-RRB- -RRB- in nest 1 -LRB- cf previous sections -RRB- . Assuming nest 1 does not exist , the number of elements fetched in nest 2 is similarly equal to S -LRB- Qgp -RRB- . However , when the two nests are considered , this assumption is not correct as far as Qfp \ Qgp < > ; . Therefore , the exact number of elements to be fetched is equal to the number of elements in Qg minus the number of elements in Qfp \ Qgp , i.e. , S -LRB- Qg -RRB- ? S -LRB- Qfp \ Qgp -RRB- . Proposition 3.3 The processor memories are assumed to be initially empty . Let T -LRB- f -LRB- : : :-RRB- -RRB- the only reference to array T within a loop nest , and T -LRB- g -LRB- : : :-RRB- -RRB- the only reference to array T within the consecutive loop nest . Then , the number of elements of array T that must be fetched in the second loop nest is equal to ST = S -LRB- Qg -RRB- ? S -LRB- Qfp \ Qgp -RRB- : 3.2.2 Cost of inter-loop communications Once the number of inter-loop communications is determined , the cost of these communications can be easily derived using the experimental results of section 2 . By extending the example of the previous section , four cases can be distinguished : ... = T -LRB- f -LRB- j1 , j2 -RRB- -RRB- ... = T -LRB- f -LRB- j1 , j2 -RRB- -RRB- T -LRB- f -LRB- j1 , j2 -RRB- -RRB- = ... T -LRB- f -LRB- j1 , j2 -RRB- -RRB- = ... ... = T -LRB- g -LRB- j1 , j2 -RRB- -RRB- T -LRB- g -LRB- j1 , j2 -RRB- -RRB- = ... ... = T -LRB- g -LRB- j1 , j2 -RRB- -RRB- T -LRB- g -LRB- j1 , j2 -RRB- -RRB- = ... Case RR Case RW Case WR Case WW In the cases RR and WR , only a copy of the data must be fetched in the second loop nest . In section 2 , an experimental method has been proposed to compute the average value of the corresponding communication time . This communication time is a function of the number of processors used . It can be denoted TR . In the cases RW and WW , the processor must request the ownership of the data along with the data itself . The corresponding average time can also be computed with the techniques of section 2 and can be denoted TW -LRB- it is also a function of the number of processors used -RRB- . Proposition 3.4 Let t be the average communication cost associated with the read/write case of references T -LRB- f -LRB- : : :-RRB- -RRB- and T -LRB- g -LRB- : : :-RRB- -RRB- . Then , the communication time due to these two references in loop nest 2 is equal to t ? ST . Note that , if the processor memories are not initially empty , the cases RW and WW must be distinguished . In the case WW , it is certain that the processor does not own the data belonging to Qfp \ Qgp , while in the case RW it is possible that the processor already owns this data . 3.2.3 General case In this section , it is assumed that loop nest 1 contains nf references to array T and that loop nest 2 contains ng references to array T . Therefore , the data elements of array T used in the first loop nest are determined by -LSB- nfi = 1Qfi , and those used in the second loop nest are determined by -LSB- ngi = 1Qgi . Then the following proposition derives directly from the proposition of section 3.2.1 Proposition 3.5 The processor memories are assumed to be initially empty . Let T -LRB- f -LRB- : : :-RRB- -RRB- the only reference to array T within a loop nest , and T -LRB- g -LRB- : : :-RRB- -RRB- the only reference to array T within the consecutive loop nest . Then , the number of elements of array T that must be fetched in the second loop nest is equal to ST = S -LRB- -LSB- nfi = 1Qfi -RRB- ? S -LRB- ? -LSB- nfi = 1Qfi ? \ ? -LSB- ngi = 1Qgi ? -RRB- : In section 3.1.3 , it was decided to approximate the number of integer points within a set by the volume of this set . While this computation is tractable for a convex polyhedron , it can be more difficult for a non-convex set . And , the union of polyhedra is not a convex polyhedron , while the intersection of polyhedra is a convex polyhedron . Therefore , the previous problem must be simplified so that only intersections of polyhedra must be computed . Theorem 3.1 Let P , Q two polyhedra . vol -LRB- P -LSB- Q -RRB- = vol -LRB- P -RRB- + vol -LRB- Q -RRB- ? vol -LRB- P \ Q -RRB- Proof The proof follows directly from geometric considerations : in the expression vol -LRB- P -RRB- + vol -LRB- Q -RRB- , P \ Q is counted twice . 2 The general formulation of the previous theorem is the following Theorem 3.2 Let -LRB- Pi -RRB- 1 < = i < = np a set of polyhedra vol -LRB- -LSB- npi = 1Pi -RRB- = X i vol -LRB- Pi -RRB- ? X i1 ; i2 ; -LRB- i1 < > i2 -RRB- vol -LRB- Pi1 \ Pi2 -RRB- + X i1 ; i2 ; i3 ; -LRB- i1 < > i2 -RRB- ; -LRB- i1 < > i3 -RRB- ; -LRB- i2 < > i3 -RRB- vol -LRB- Pi1 \ Pi2 \ Pi3 -RRB- ? : : : Proof The proof follows by induction from the previous theorem . 2 Using this last theorem , ST can be computed as the sum and differences of intersections of polyhedra . So , using all the above results , the following observation can now be stated Observation 3.1 It is possible to compute the number and cost of communications between two consecutive loop nests associated with each array within these loop nests . 3.2.4 Further issues Multiple loop nests When more than two loop nests are used in a subroutine , two methods can be used . The most simple one is to assume that only the communications between two consecutive loop nests are significant , i.e. , that the memory content due to the second previous loop nest can be neglected . The second method is to assume that , for each array T , there is a set of elements of T in the processor memory . However , it can not be assumed that this set of elements is a polyhedron . However , this set can probably be described as the union of several polyhedra -LSB- PT . This set of polyhedra must be split into two parts : the data owned , -LSB- P oT , and not owned , -LSB- P no T , by the processor . If -LSB- PT = -LRB- -LSB- P oT -RRB- -LSB- -LRB- -LSB- P no T -RRB- is known , then the results of section 3.2.3 can be extended by replacing -LSB- ngi = 1Qgi with -LRB- -LSB- ngi = 1Qgi -RRB- -LSB- -LRB- -LSB- PT -RRB- . With respect to computing the communication costs , -LSB- P oT and -LSB- P no T should be distinguished . Intra-loop communications In the previous sections , it has been assumed that no intra-loop communication takes place in the first loop nest , so that the data distributed to each processor at the beginning of the loop nest is present in the processor memory at the end of the loop nest . Two cases can flaw this assumption . First , loop nest 1 contains dependences even though it is parallelized -LRB- using the wavefront method for instance -RRB- . In this case , the data move between processors in a predictable manner . Extending the framework to this case is tractable and this problem will be addressed in future versions of this work . Second , false sharing occurs in loop nest 1 . The problem is then to determine which is the last processor to touch a given page . Even though this problem will also be addressed , it is not obvious that a solution can be derived easily . Still , it shows that false sharing and data distribution should preferably not be studied separately . Reads/Writes In the previous sections , it has been assumed that only one type of communications -LRB- either RR/WR or RW/WW -RRB- can take place between the two loop nests . However , it is possible that some data can be accessed with a RR/WR communication type , while other data accessed with a RW/WW communication type . Since TW > TR , we can write TW = TR + ? W . Let us consider the following extension of the example of section 3.2.1 where all possible cases occur . c Nest 1 DOALL j1 = 1 , N DO j2 = 1 , N ... = T -LRB- f1 -LRB- j1 , j2 -RRB- -RRB- T -LRB- f2 -LRB- j1 , j2 -RRB- -RRB- = ... ENDDO ENDDO c Nest 2 DOALL j1 = 1 , N DO j2 = 1 , N ... = T -LRB- g1 -LRB- j1 , j2 -RRB- -RRB- T -LRB- g2 -LRB- j1 , j2 -RRB- -RRB- = ... ENDDO ENDDO The number of communications is equal to ST = S -LRB- Qrg -LSB- Qwg -RRB- ? S -LRB- -LRB- Qrf -LSB- Qwf -RRB- \ -LRB- Qrg -LSB- Qwg -RRB- -RRB- , where Qr ? correspond to the polyhedra of data read and Qw ? correspond to the polyhedra of data written . Therefore , the expanded expression of ST -LRB- cf section 3.2.1 -RRB- is the following ST = S -LRB- Qrg -RRB- + S -LRB- Qwg -RRB- ? S -LRB- Qrg \ Qwg -RRB- ? ? S -LRB- Qrf \ Qrg -RRB- + S -LRB- Qrf \ Qwg -RRB- + S -LRB- Qwf \ Qrg -RRB- + S -LRB- Qwf \ Qwg -RRB- ? S -LRB- Qrf \ Qrg \ Qwg -RRB- ? S -LRB- Qwf \ Qrg \ Qwg -RRB- ? S -LRB- Qrf \ Qwf \ Qrg -RRB- ? S -LRB- Qrf \ Qwf \ Qwg -RRB- + S -LRB- Qrf \ Qwf \ Qrg \ Qwg -RRB- ? And , by distinguishing the cases RR/RW , WR/WR , the communication cost can be derived Tcost = TR ? S -LRB- Qrg -RRB- + TW ? S -LRB- Qwg -RRB- ? TW ? S -LRB- Qrg \ Qwg -RRB- ? ? TR ? S -LRB- Qrf \ Qrg -RRB- + TW ? S -LRB- Qrf \ Qwg -RRB- + TR ? S -LRB- Qwf \ Qrg -RRB- + TW ? S -LRB- Qwf \ Qwg -RRB- ? TW ? S -LRB- Qrf \ Qrg \ Qwg -RRB- ? TW ? S -LRB- Qwf \ Qrg \ Qwg -RRB- ? TR ? S -LRB- Qrf \ Qwf \ Qrg -RRB- ? TW ? S -LRB- Qrf \ Qwf \ Qwg -RRB- + TW ? S -LRB- Qrf \ Qwf \ Qrg \ Qwg -RRB- ? This method can be extended straightforwardly to the general case . Page size Up to now , the granularity of a communication was assumed to be a data element . In a multiprocessor , the coherence is enforced at the page or cache line level -LRB- for the KSR , at the subpage level , i.e. , 128 bytes -RRB- . This parameter strongly affects the number of communications actually taking place , but it is difficult to compute the actual number of pages touched by any reference . A reasonable approximation is to define a cache line access stride for each reference and multiply the data element volume by this stride in order to obtain a cache line volume . Let T -LRB- f -LRB- j1 ; : : : ; jn -RRB- a reference and T -LRB- P i = 1 ; r OEiji + OE0 -RRB- the linearized version of this reference . The access stride s can be defined as s = min -LRB- min i = 1 ; r -LRB- OEi -RRB- ; PS -RRB- , where PS is the page size . So the S function must be changed into S -LRB- Qf -RRB- = sf ? vol -LRB- Qf -RRB- where Qf ; sf are respectively a polyhedron and the access stride associated with f . A more general definition can also be provided S -LRB- Qf1 \ Qf2 \ : : : Qft -RRB- = min -LRB- sf1 ; sf2 ; : : : ; sft -RRB- ? vol -LRB- Qf1 \ Qf2 \ : : : Qft -RRB- : 3.3 Applications The purpose of this section is to highlight the possible applications of this framework , in addition to performance analysis of multiprocessors memory system which is the initial goal of this work . 3.3.1 Parallelization Up to now , parallelization strategies are mostly concerned with optmizing the performance of a given loop nest by finding the optimal direction of parallelization and the optimal scheduling of the iterations for that loop nest . However , due to the fine grain of many numerical loops , parallelizing at the loop nest level is not optimal . There is a necessity for taking into account data movements accross several consecutive loop nests and for parallelizing such loop nests in order to maximize their cache affinity . Heuristics have been proposed for improving cache affinity so as to reduce coherence costs -LSB- 10 -RSB- , but there is no framework for modeling inter-loop communications like those developped for intra-loop communications -LSB- 6 , 11 , 12 -RSB- . Within the framework defined above , the total communication time Tinter between l loop nests can be computed symbolically . The directions of parallelization d1 ; : : : ; dl of the l consecutive loop nests can be considered as parameters of this expression . In terms of inter-loop communications , near-optimal performance would be obtained for -LRB- d1 ; : : : ; dl -RRB- = max d1 ; : : : ; dl Tinter -LRB- d1 ; : : : ; dl -RRB- : Using the proposed framework , this problem can be intuitively phrased as maximizing the sum of the size of the intersections between all the polyhedra associated with each loop nest and each array . 3.3.2 Software coherence Software-managed coherence has been proposed as an inexpensive solution to cache coherence -LSB- 13 -RSB- . One of the main issues of cache coherence is to identify the data elements shared by two consecutive loop nests in order to invalidate them . The lack of efficient techniques for precisely identifying shared elements result in excessive invalidations and performance degradations . The proposed framework can be used to precisely identify the shared elements and to perform the minimum number of invalidations , i.e. , to enhance the selectivity of current software-coherence strategies . 3.3.3 Inter-loop locality In a uniprocessor environment , the proposed framework can be used to evaluate the inter-loop locality . In addition to performance evaluation , it can be used to precisely evaluate the number of data elements that are referenced in a loop nest , and that were not referenced by the previous loop nest , in order to accurately drive a software-controlled prefetching strategy . 4 Conclusions In this paper , the problem of evaluating the communication time in a multiprocessor system has been addressed for numerical codes . A set of tiny programs has been proposed for extracting the main characteristics of the multiprocessor memory system . This systematic benchmarking had two goals : highlight the multiprocessor memory behavior in a number of classic cases , and quantify the cost of communications for the different communication patterns that can be found in a numerical code . Then , the problem of evaluating the performance of the memory system is formally addressed . Counting the number of cache misses -LRB- capacity and conflict -RRB- occuring during execution of a loop nest can be considered as a uniprocessor problem and has been previously addressed . In a multiprocessor system , one of the main issues are the compulsory misses , i.e. , the first reference to data , because they can correspond to inter-processor communications . Therefore , a formal framework has been proposed for computing the number of first references -LRB- i.e. , compulsory misses -RRB- that result in communications . The problem actually sums up to computing the number of data elements used together by the loop nest considered and the previous loop nests . It is shown that computing this intersection is a tractable task for any number of references within the two loop nests and for any direction of parallelization . This work has applications in the domain of software-controlled coherence , the optimization of parallelization at the subroutine level -LRB- i.e. , over several loop nests -RRB- and for the exploitation of uniprocessor inter-loop locality . References -LSB- 1 -RSB- O. Temam , C. Fricker , and W. Jalby . Cache Interference Phenomena . In Proceedings of the ACM SIGMETRICS Conference on Measurement and Modeling of Computer Systems , 1994 . -LSB- 2 -RSB- Mark Hill . Aspects of Cache Memory and Instruction Buffer Performance . PhD thesis , University of California Berkeley , 1987 . -LSB- 3 -RSB- Monica Lam , Edward E. Rothberg , and Michael E. Wolf . The Cache Performance of Blocked Algorithms . In Fourth International Conference on Architectural Support for Programming Languages and Operating Systems , April 1991 . -LSB- 4 -RSB- George Cybenko , Lyle Kipp , Lynn Pointer , and David Kuck . Supercomputing Performance Evaluation and the Perfect Benchmarks . In Supercomputing ' 90 , pages 254 -LCB- 266 , 1990 . -LSB- 5 -RSB- Thierry Montaut . PhD Thesis . PhD thesis , University of Rennes , to appear in 1994 . -LSB- 6 -RSB- Christine Eisenbeis , William Jalby , Daniel Windheiser , and Francois Bodin . A Strategy for Array Management in Local Memory . In Proceedings of the Third Workshop on Programming Languages and Compilers for Parallel Computing , 1990 . -LSB- 7 -RSB- A. Schrijver . Theory of Linear and Integer Programming . Wiley-Interscience , 1986 . -LSB- 8 -RSB- She Zhiyu , Zhiyuaun Li , and Pen-Chung Yew . An Empirical Study on Array Subscripts and Data Dependendencies . Technical Report 840 , Center for Supercomputing Research and Development , University of Illinois at Urbana-Champaign , August 1989 . -LSB- 9 -RSB- C. Eisenbeis , O. Temam , and H. Wijshoff . On Efficiently Characterizing the Solutions of Linear Diophantine Equations and Its Application to Data Dependence Analysis . Technical report , Department of Computer Science , Technical Report RUU-CS-92-01 , Utrecht University , January 1992 . -LSB- 10 -RSB- H. Li and K. C. Sevcik . Exploiting Cache Affinity in Software Cache Coherence . In Proceedings of the ACM International Conference on Supercomputing , 1994 . -LSB- 11 -RSB- Kathryn S. McKinley . Automatic and Interactive Parallelization . PhD thesis , Rice University , Technical Report CRPC-TR92214 , April 1992 . -LSB- 12 -RSB- Michael Wolf and Monica Lam . A Data Locality Optimizing Algorithm . In Proceedings of the ACM SIGPLAN '91 Conference on Programming Language Design and Implementation , volume 26 -LRB- 6 -RRB- , pages 30 -LCB- 44 , June 1991 . -LSB- 13 -RSB- Hoichi Cheong and Alexander V. Veidenbaum . Compiler-Directed Cache Management in Multiprocessors . IEEE Computer , 23 -LRB- 6 -RRB- :39 -LCB- 47 , June 1990 .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O