LIPLaboratoire de l'Informatique du Parall?lisme Ecole Normale Sup?rieure de Lyon Unit ? de recherche associ?e au CNRS n ? 1398 Computing over the Reals with Addition and Order Pascal Koiran August 24 , 1993 Research Report No 93-25 Ecole Normale Sup?rieure de Lyon 46 , All?e d'Italie , 69364 Lyon Cedex 07 , France , T?l?phone : + 33 72 72 80 00 ; T?l?copieur : + 33 72 72 80 80 ; Adresses ? lectroniques : lip@frensl61.bitnet; lip@lip.ens-lyon.fr -LRB- uucp -RRB- . Computing over the Reals with Addition and Order Pascal Koiran August 24 , 1993 Abstract We provide a fairly complete picture of the complexity theory of additive real machines . This model of computation is a restriction of the real Turing machine of Blum , Shub & Smale , since addition and subtraction are the only legal arithmetic operations . Removing the order relation < on R yields an even weaker class of machines , which is also studied . Our main results are : ffl characterizations of the classes of recognizable boolean languages ; ffl equivalence of real and digital nondeterminism ; ffl a simpler proof of Meer 's Plin < > NPlin result . R?esum?e Nous fournissons une description assez compl?ete de la th?eorie de la complexit?e des machines r?eelles additives . Ces machines sont des variantes affaiblies de la machine de Turing r?eelle de Blum , Shub & Smale , car l'addition et la soustraction sont les seules op?erations arithm?etiques l?egales . En supprimant la relation d'ordre < sur R , on obtient une classe de machines encore moins puissantes , qui est ? etudi?ee ? egalement . Nos principaux r?esultats sont : ffl des caract?erisations des classes de langages bool?eens reconnaissables ; ffl l' ? equivalence du non-d ? eterminisme r?eel et du non-d ? eterminisme digital ; ffl une preuve plus simple du r?esultat Plin 6 = NPlin de Meer . Mots-cl ? es : complexit?e , machines r?eelles , poly?edres . 1 Introduction The real Turing machine of Blum , Shub & Smale -LSB- 2 -RSB- is endowed with the four basic arithmetic operations -LRB- addition , subtraction , multiplication , division -RRB- and with the order relation < on R . It seems however that multiplication and division are of little help for a number of interesting problems such as the real-valued generalizations of the Traveling Salesman Problem or the Knapsack Problem . In order to study these problems , it is quite natural to work with real machines equipped with addition and subtraction only . Let P < add and NP < add be the class of real languages -LRB- subsets of R1 = S1n = 1 Rn -RRB- that can be recognized in deterministic and non-deterministic polynomial time in this additive BSS model . It is possible to make fruitful comparisons with discrete complexity theory if we consider real machines with discrete inputs . In this paper we show that : ffl P < add \ ?? = P = poly ; ffl NP < add \ ?? = NP = poly . Here ? = f0 ; 1g is the binary alphabet and we use the notation C \ ?? = fL \ ?? ; L 2 Cg where C is a class of real languages . The classes P/poly and NP/poly are the nonuniform counterparts of P and NP -LRB- see e.g. -LSB- 1 -RSB- for more details -RRB- . We also consider a model in which no order relation is available -LRB- hence equality is the only test allowed -RRB- . In this case real machines are not more powerful than Turing machines for discrete inputs , i.e. , ffl P = add \ ?? = P ; ffl NP = add \ ?? = NP . The consideration of the real Traveling Salesman Problem or the real Knapsack Problem suggests a further restriction of the standard BSS model , this time on the type of non-determinism allowed . In the standard BSS model a guess is a sequence of real numbers , but the natural non-deterministic algorithms for these two problems make guesses that are of a combinatorial nature -LRB- a permutation of the sequence of cities or a subset of the set of objects -RRB- . It is therefore quite natural to study digital nondeterministic -LRB- or weak -RRB- real machines whose guesses are sequence of 0 's and 1 's , like those of Turing machines . We prove that digital non-determinism is in fact equivalent to the standard real determinism . More precisely , ffl NDP < add = NP < add ; ffl NDP = add = NP = add . where NDP denotes weak non-deterministic classes . Additive machines and weak non-determinism have already been studied in -LSB- 3 -RSB- . The weak BSS model -LSB- 4 -RSB- and linear machines -LSB- 6 -RSB- are two other related models . Their computational power lies in-between those of the standard BSS model and of additive machines . In section 2 we shall give a simpler proof of Meer 's Plin < > NPlin result , from which it follows that P = add 6 = NP = add . The remainder of the paper is organized as follows . In section 2 we introduce a few general notations and study real machines without order . In section 3 we establish a few properties of polyhedra that are useful for studying real machines with order -LRB- a polyhedron is a subset of Rn defined by a system of large or strict linear inequalities having integral coefficients -RRB- . In particular , we give two purely geometrical results : ffl a non-empty polyhedron defined by a system of \ small '' inequalities contains a \ small '' rational point ; ffl A projection of a polyhedron defined by a system of \ small '' inequalities can be defined by a system of \ small '' inequalities . Real machines with order are discussed in section 4 . 2 Computing over the reals with addition and equality Throughout the paper , = -LRB- 1 ; : : : ; p -RRB- is the vector of constants occurring in the program of a real machine -LRB- we assume without loss of generality that one of these constants is equal to 1 -RRB- . The size s -LRB- n -RRB- of an integer n is the number of digits of its radix-2 representation , and the size of a rational number x = p = q is the minimum value of s -LRB- p -RRB- + s -LRB- q -RRB- where the integers p and q are such x = p = q . We assume w.l.o.g. that all tests are of the form \ x ? = 0 '' . Theorem 1 is in fact a special case of a more general result on a model in which a moderate usage of multiplication and division is allowed -LSB- 4 -RSB- . The general result relies on Gr?obner bases or the primitive element theorem , but here some very elementary linear algebra is sufficient . Theorem 1 P = add \ ?? = P. Proof . Clearly P ae P = add \ ?? . We prove the converse below . At any time in the computation of real machine on an input u 2 f0 ; 1g + , the content of a variable x is of the form x = pX i = 1 ai i where the ai 's are integers of polynomial size . Let E be the vector space over Q spanned by 1 ; : : : ; p and let m be the dimension of E . We can assume without loss of generality that -LRB- 1 ; : : : ; m -RRB- is a base of E . Let -LRB- bij -RRB- be the matrix of rational coefficients such that i = m X j = 1 bij j -LRB- 1 < = i < = p -RRB- : Since x = m X j = 1 pX i = 1 aibij ! j ; the test x ? = can be performed formally by checking that Ppi = 1 aibij = for 1 < = j < = m. ? We now turn to the study of digital non-determinism . Theorem 2 NDP = add = NP = add . The proof of this result relies on the following lemma . Lemma 1 Let S be a system of -LRB- dis -RRB- equations on y 2 Rn of the form 8 >> >> < >> >> : nX j = 1 aijyj = bi -LRB- 1 < = i < = p -RRB- nX j = 1 cijyj < > di -LRB- 1 < = i < = q -RRB- with aij ; cij 2 N and bi ; di 2 R. Let us define the size of S as the number of real constants in the system plus the number of bits needed to represent the integral constants . It is possible to decide in P = add whether S is satisfiable . Proof . We first solve the system of equalities by Gaussian elimination . If this system is not satisfiable , S is of course not satisfiable . If it is satisfiable , we obtain -LRB- up to a permutation of indices -RRB- a parametrization of the solution space of the form yi = ei + nX j = k +1 fijyj -LRB- 1 < = i < = k -RRB- where yk +1 ; : : : ; yn are free variables -LRB- possibly , k = n -RRB- . Next we substitute this expression in the inequalities to look for a contradiction . If we obtain an inequality of the form di < > di , S is of course not satisfiable . Otherwise S is satisfiable , since Rn?k ? 1 can not be equal to the union of a finite number of hyperplanes . It remains to show that this algorithm runs in polynomial time in the additive BSS model . Note that there are no multiplications or divisions between real variables in this algorithm . It is easy to multiply a real number y by an integer x = x1 : : : xm using a sequence of \ shift-and-add '' operations , e.g. , product : = 0 ; for i : = 1 to m do if x -LSB- i -RSB- = 0 then product : = 2 * product else product : = 2 * product + y . If we have to divide y by x , we simply represent the quotient by the pair -LRB- y ; x -RRB- . The overall running time will be polynomial if the integers occurring during the execution of the algorithm are of polynomial size . This property holds for the Gaussian elimination procedure -LRB- -LSB- 7 -RSB- , Theorem 3.3 -RRB- . It is not difficult to see that there can not be any blowup in integer size in the rest of the algorithm . ? Note that for Theorem 2 it would be sufficient to show that the satisfiability of S is in NP = add . Proof of Theorem 2 . Clearly NDP = add ae NP = add : a digital guess g 2 f0 ; 1gnO -LRB- 1 -RRB- can be computed by comparing to zero each component of a real guess h 2 RnO -LRB- 1 -RRB- . We prove the converse below . Let M be a non-deterministic machine recognizing L 2 NP = add . We construct a digitally non-deterministic machine M0 recognizing L in polynomial time . On an input u 2 Rn , this machine first guesses the results of all tests in the program . Since M runs in polynomial time , this needs to be done for a time T -LRB- n -RRB- which is polynomial in n. Adding dummy instructions if necessary , we can assume that all computations of M halt exactly in time T -LRB- n -RRB- . During a computation the content of any variable is always a linear function of the constants , the input u 2 Rn and the real guess y 2 Rp -LRB- n -RRB- of M. Hence for a given sequence g of test results , the constants , inputs x and real guesses y corresponding to accepting computations are defined by a system Sg -LRB- ff ; x ; y -RRB- of linear equations and disequations -LRB- these constraints ensure that the computation is accepting and that the results of all tests are given by g -RRB- . For a fixed , an input u is recognized if and only if 9g 9y Sg -LRB- ff ; u ; y -RRB- is satisfied : After g has been guessed , the existence of y such that Sg -LRB- ff ; u ; y -RRB- is satisfied can be tested by the algorithm of Lemma 1 . This takes polynomial time since the integral constants in the system are of polynomial size . ? The non-deterministic counterpart of Theorem 1 is a simple consequence of Theorem 2 . Theorem 3 NP = add \ ?? = NP . Proof . Clearly NP ae NP = add . For the conserve we just have to show that NDP = add ae NP . For a fixed digital guess , M behaves like a deterministic real machine . More precisely , deciding whether a pair -LRB- u ; y -RRB- 2 f0 ; 1gn ? f0 ; 1gnO -LRB- 1 -RRB- corresponds to an accepting computation of M is a P = add problem . According to Theorem 1 this problem is actually in P , whence the result . ? In the rest of this section we turn our attention to the more powerful model of linear machines . A linear machine can perform computations of the form y m X i = 1 cixi + b ; where b and the ci 's are constants of the machine . Equality is still the only comparator allowed . Meer proved a P6 = NP result for this model in -LSB- 6 -RSB- -LRB- Plin < > NPlin with his notations -RRB- . We now give a simpler proof of his result . Theorem 4 The following problem L is in NPlin but not in Plin : given x 2 Rn , is there ffl 2 f0 ; 1gn such that Pni = 1 fflixi = 1 ? Proof . The problem is clearly in NPlin . Let us assume that there exists a linear machine M which recognizes L in polynomial time . All tests performed by M are of the form l -LRB- x -RRB- ? = 0 , where l is an affine function of the input x . Since M can perform only a finite number of tests in finite time , there exists some input x62L such that the answers to the tests l1 -LRB- x -RRB- ? = ; : : : ; lm -LRB- x -RRB- ? = of M on this input are all negative . Since M runs in polynomial time , at least one of the 2n tests of the form Pni = 1 ffli xi ? = 1 has not been performed . Consider such an ffl and an input y 2 Rn such that Pni = 1 ffli yi = 1 ; l1 -LRB- y -RRB- 6 = 0 ; : : : ; l1 -LRB- y -RRB- 6 = 0 -LRB- y exists since the m last constraints remove only a -LRB- n ? 2 -RRB- - dimensional subspace from the hyperplane satisfying the first constraint -RRB- . The outcomes of all tests are the same for y as for x , hence M declares that y62L since x62L . This is a contradiction , since in fact y 2 L. ? A similar result was proved by Meer for a multidimensional version of L . Since L is clearly in NP = add , it follows from Theorem 4 that P = add 6 = NP = add . 3 Some basic facts on polyhedra In this section we show that : ffl a non-empty polyhedron defined by a system of \ small '' inequalities contains a \ small '' rational point -LRB- Theorem 6 -RRB- ; ffl The satisfiability problem for a system of linear inequalities is in NP < add . ffl A projection of a polyhedron defined by a system of \ small '' inequalities can be defined by a system of \ small '' inequalities -LRB- Theorem 8 -RRB- . The following linear programming result will be useful -LRB- see -LSB- 7 -RSB- , corollary 7.1 i -RRB- . Lemma 2 -LRB- Caratheodory 's theorem -RRB- If a vector b 2 Rn is a positive linear combinations of vectors a1 ; : : : ; am 2 Rn -LRB- i.e. , if there exist coefficients xi > = such that b = Pmi = 1 xiai -RRB- , then there exists a linearly independent subset fai1 ; : : : ; aikg ae fa1 ; : : : ; amg such that b is a positive linear combination of ai1 ; : : : ; aik . We first establish the existence of \ small '' points for the case of closed polyhedra . Theorem 5 is a well-known result ; the proof that we give here is due to W. Maass -LSB- 5 -RSB- and its principle will be used later in this section . Theorem 5 Let P be a polyhedra of Rn defined by a system Ax < = b of N linear inequalities , where the entries of A and b are integers of size L . If P < > ; , there exists a rational point x 2 P of size polynomial in L and n -LRB- i.e. , s -LRB- x -RRB- = -LRB- Ln -RRB- O -LRB- 1 -RRB- -RRB- . Proof . The original system can be transformed into a system of the form A0x0 = b ; x0 > = -LRB- 1 -RRB- by writing each variable xi as the difference of two non-negative variables , and adding a \ slack variable '' to each inequality . In the N ? -LRB- 2n + N -RRB- matrix A0 , the N columns for the N slack variables of x0 form an idendity matrix . Hence A0 has rank N . By hypothesis , -LRB- 1 -RRB- has a solution . It follows from Caratheodory 's theorem that there exists also a satisfiable system of the form A00x00 = b ; x00 > = -LRB- 2 -RRB- where A00 is a N ? N matrix made of N linearly independent columns of A0 . Since A00 has full rank , -LRB- 2 -RRB- has in fact a unique solution which is given by Cramer 's rule : x00j = det -LRB- A00j -RRB- = det -LRB- A00 -RRB- for j = 1 ; : : : ; N , where A00j results from A00 by replacing its jth column by b . Since all but at most 2n columns of A00 contain exactly one 1 and else only 0 's , one can bring by permutations of rows and columns each of the matrices A00 ; A00j into the form B = C D I ! where C is a square matrix of size 2n + 1 . Hence the determinant of B is an integer of absolute value smaller than -LRB- 2n + 1 -RRB- ! 2L -LRB- 2n +1 -RRB- . ? Theorem 6 Let P be a polyhedra of Rn defined by a system S of N strict or large inequalities of the form Ax < = b ; A0x < b0 where the entries of A ; A0 ; b and b0 are integers of size L . If P < > ; , there exists a rational point x 2 P of size polynomial in L and n -LRB- i.e. , s -LRB- x -RRB- = -LRB- Ln -RRB- O -LRB- 1 -RRB- -RRB- . Proof . Let P be the polyhedron of Rn +1 defined by the system S0 obtained from S by the following transformations : ffl add a new variable K and the constraint K > = 1 ; ffl replace the subsystem Ax < = b by Ay < = Kb ; ffl replace the subsystem Ax < b by Ay < = Kb ? 1 . Clearly P < > ; if and only if P < > ; . If P < > ; , P contains a small point -LRB- K ; y -RRB- according to Theorem 5 , and the transformation xi = yi = K yields a small point in P . ? Note that the number N of inequations in the system does not come into play in the bound . This point is crucial for the applications of this result in sections 2 and 4 . Theorem 7 Let S be a system of N inequations on y 2 Rn of the form Ax < = b ; A0x < b0 with aij ; a0ij 2 N and bi ; b0i 2 R. Let us define the size of S as the number of real constants in the system plus the number of bits needed to represent the integral constants . It is possible to decide in NDP < add whether S is satisfiable . Proof . We transform S into a new system S0 of the form Cz < = d , like in the proof of Theorem 6 -LRB- z = -LRB- K ; y -RRB- with K 2 R and y 2 Rn -RRB- . As we noticed before , S is satisfiable if and only if S0 is satisfiable . The satisfiability of S0 can be checked with the proof technique of Theorem 5 : we transform S0 into an equivalent system of the form C0z0 = d ; z0 > = -LRB- 3 -RRB- by adding a slack variable to each inequality and writing each variable yi as the difference of two non-negative variables -LRB- this needs not be done for the variable K since the constraint K > = 1 already appears in the system -RRB- . In the N ? -LRB- N + 2n + 1 -RRB- matrix C0 , the N columns for the N slack variables of z0 form an idendity matrix . Hence C0 has rank N . It follows from Caratheodory 's theorem that -LRB- 3 -RRB- is satisfiable if and only if there exists also a satisfiable system of the form C00z00 = d ; z00 > = -LRB- 4 -RRB- where C00 is a N ? N matrix made of N linearly independent columns of C0 . Since C00 has full rank , -LRB- 4 -RRB- has in fact a unique solution which is given by Cramer 's rule : z00j = det -LRB- C00j -RRB- = det -LRB- C00 -RRB- for j = 1 ; : : : ; N , where C00j results from C00 by replacing its jth column by d . The overall algorithm is therefore quite simple : 1 . guess non-deterministically a N ? N submatrix C00 of C0 ; 2 . check that det -LRB- C00 -RRB- 6 = 0 ; 3 . check that z00j = det -LRB- C00j -RRB- = det -LRB- C00 -RRB- > = 0 , for j = 1 ; : : : ; N . It remains to show that this can be done in polynomial time in the additive BSS model . Obviously the last step does not actually require a division : it suffices to check that sign -LRB- det -LRB- C00j -RRB- -RRB- : sign -LRB- det -LRB- C00 -RRB- -RRB- > = 0 . Computing the determinants is a little more difficult due to the presence of real numbers in the system . Note however that these real numbers bi occur in a single column of C00 -LRB- and in at most one column of C00j -RRB- . In order to compute det -LRB- C00 -RRB- -LRB- or det -LRB- C00j -RRB- -RRB- , it is thus possible to perform an expansion along this column . The N -LRB- integral -RRB- subdeterminants can be computed by Gaussian elimination and the N integer?real products by shift-and-add operations . This takes polynomial time -LRB- see Lemma 1 -RRB- . ? The methods of theorems 5 and 7 can be used to prove the following result : a projection of a polyhedron defined by \ small '' inequations can be described by small inequations . We give this result here because of its interest from a geometrical and logical point of view , but it will not be used in the rest of the paper . For the special case of closed polyhedra , this result was obtained previously in -LSB- 8 -RSB- by different methods . Theorem 8 Let P be a polyhedra of Rn defined by a system of inequalities of the form Ax < = b ; A0x < b0 where the entries of A ; A0 ; b and b0 are integers of size L . The image of P by any projection of the form -LRB- x1 ; : : : ; xm ; : : : ; xn -RRB- 7 ! -LRB- x1 ; : : : ; xm -RRB- is a polyhedron of Rm defined by a system of inequalities whose coefficients are of size -LRB- Ln -RRB- O -LRB- 1 -RRB- . Proof Sketch . We will not obtain directly a system of inequalities describing the projected polyhedron P , but a union of systems -LRB- this amounts to giving a polyhedral decomposition of P -RRB- . A single system for P can be obtained by picking some inequalities in each system of the union . By definition a point y = -LRB- x1 ; : : : ; xm -RRB- belongs to P iff there exists a point z = -LRB- xm +1 ; : : : ; xn -RRB- such that -LRB- y ; z -RRB- 2 P , or in other words if the polyhedra of Rn?m defined by the system Bz < = -LRB- y -RRB- ; B0z < -LRB- y -RRB- -LRB- 5 -RRB- is non-empty . Here B and B0 are the matrices made of the n?m last columns of A and A0 , i -LRB- y -RRB- = bi ? Pmj = 1 aijxj and i -LRB- y -RRB- = b0i ? Pmj = 1 a0ijxj . Let us apply the transformation of Theorem 7 : this condition is equivalent to the existence of a constant K > such that a certain -LRB- n ? m -RRB- - dimensional system S of the form n?m X j = 1 bijuj < = K i -LRB- y -RRB- ? ffli is satisfiable -LRB- ffli 2 f0 ; 1g -RRB- . The proof technique of Theorem 5 shows that the satisfiability of S is equivalent to the satisfiability of a least one of a finite number of systems of the form Kli -LRB- y -RRB- > = ci ; i = 1 ; : : : ; N -LRB- 6 -RRB- where li is an affine function . The permutation argument of Theorem 5 shows that the coefficients of such a system are of polynomial size -LRB- in n and L -RRB- . It is not difficult to see that the set of y 's such that -LRB- 6 -RRB- is satisfied for some K > can be described by a finite set of linear systems -LRB- whose coefficients have the required size -RRB- . The details are left to the reader . ? From a logical point of view , the fact that projections of polyhedra are still polyhedra can be restated as \ an existential formula in the theory of the reals with addition and order is equivalent to a quantifier-free formula '' . Theorem 8 asserts that this quantifier elimination does not produce any blowup in coefficient size . 4 Computing over the reals with addition and order The following theorem is a special case of the PWRD = P = poly result of -LSB- 4 -RSB- . Here we can give a simpler proof than in the general case . Theorem 9 P < add \ ?? = P = poly . Proof . In order to show that P = poly ae P < add \ ?? , we encode an advice function in the digits ai of a real constant = : a1a2 : : : ai : : : . Let us choose an encoding scheme such that infinitely many ai 's are equal to 0 . The value of a1 can be easily computed since a1 = , < 1 = 2 . In order to compute a2 , we shift to the left with the instruction : = 2 ? a1 . Now a2 can be computed like a1 . By iterating this process , it is possible to retrieve as many digits of c as desired . Since the advice are of polynomial size , the whole decoding process takes polynomial time . In order to prove the converse , we replace for each input size n the real constants by rational approximations such that the outcomes of all tests in the program are unchanged . This constraint gives a system of -LRB- exponentially many -RRB- linear equalities and inequalities with coefficients of polynomial size . According to Theorem 6 , there exists a solution of polynomial size . ? Theorem 10 NP < add = NDP < add . Proof . The proof is identical to that of Theorem 2 , except that we obtain a system Sg of linear equalities and inequalities . The satisfiability of this system can be tested with the non-deterministic algorithm of Theorem 7 . ? Theorem 11 NP < add \ ?? = NP = poly . Proof . Clearly NP = poly ae NP < add \ ?? -LRB- see the proofs of theorems 3 and 9 -RRB- . According to Theorem 10 , it suffices to show that NDP < add \ ?? ae NP = poly to prove the converse . For a fixed digital guess , M behaves like a deterministic real machine . More precisely , deciding whether a pair -LRB- u ; y -RRB- 2 f0 ; 1gn ? f0 ; 1gnO -LRB- 1 -RRB- corresponds to an accepting computation of M is a P < add problem . According to Theorem 9 this problem is actually in P/poly , whence the result . ? Acknowledgment I would like to thank Bruno Poizat for several useful suggestions . References -LSB- 1 -RSB- J.L. Balc?azar , J. Di?az , and J. Gabarr?o . Structural Complexity I. EATCS Monographs on Theoretical Computer Science . Springer-Verlag , 1988 . -LSB- 2 -RSB- L. Blum , M. Shub , and S. Smale . On a theory of computation and complexity over the real numbers : NP-completeness , recursive functions and universal machines . Bulletin of the American Mathematical Society , 21 -LRB- 1 -RRB- :1 -LCB- 46 , July 1989 . -LSB- 3 -RSB- F Cucker and M. Matamala . On weak nondeterminism . preprint , 1993 . -LSB- 4 -RSB- P. Koiran . A weak version of the Blum , Shub & Smale model . In Proc . 34th FOCS Symp. , 1993 . to appear . -LSB- 5 -RSB- W. Maass . Bounds for the computational power and learning complexity of analog neural nets . In Proc . 25th STOC , pages 335 -LCB- 344 , 1993 . -LSB- 6 -RSB- K. Meer . A note on a P < > NP result for a restricted class of real machines . Journal of Complexity , 8:451 -LCB- 453 , 1992 . -LSB- 7 -RSB- A. Schrijver . Theory of Linear and Integer Programming . Wiley , New-York , 1986 . -LSB- 8 -RSB- E. D. Sontag . Real addition and the polynomial hierarchy . Information Processing Letters , 20:115 -LCB- 120 , 1985 .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


University of Leeds SCHOOL OF COMPUTER STUDIES RESEARCH REPORT SERIES Report 94.34 New Methods for the Initialisation of Clusters by Mohammed Al-Daoud & Stuart Roberts Division of Operational Research and Information Systems December 1994 Abstract : One of the most widely used clustering techniques is the kmeans algorithm . Solutions obtained from this technique are dependent on the initialisation of cluster centres . In this article , two initialisation methods are developed . These methods are particularly suited to problems involving very large data sets . The methods have been applied to different data sets and good results are obtained . 1 Introduction Clustering techniques have received attention in many areas , such as engineering , medicine , biology and computer science . The purpose of clustering is to group together data points which are close to one another . The k-means algorithm -LRB- MacQueen , 1967 -RRB- is one of the most widely used techniques for clustering . Most of the vector quantization techniques , for example , are based on the k-means method -LRB- Gersho , 1992 -RRB- , which being a heuristic method , yields a solution that is not guaranteed to be optimal , but which depends on the choice of the initial cluster centres -LRB- Babu , 1993 ; Nasrabadi , 1988 -RRB- . Two simple approaches to cluster centre initialisation are either to select the initial values randomly , or to choose the first K samples of the data points -LRB- MacQueen , 1967 -RRB- . As an alternative , different sets of initial values are chosen -LRB- out of the data points -RRB- and the set which is closest to optimal 1 is chosen . However , testing different initial sets is considered impracticable criteria , especially for large number of clusters -LRB- Ismail , 1989 -RRB- . For example , the number of complete enumerations of every possible partition of 50 data points into five clusters is 7.4 x 1032 -LRB- Kaufman , 1990 , p. 115 -RRB- . -LRB- Linde , 1980 -RRB- proposed the Binary Splitting -LRB- BS -RRB- technique to initialise the clusters . The BS algorithm is as follows : 1 . Calculate the value of the centroids of the data points , ci , and assign this value as the initial centre for the first cluster ; 2 . Double the number of the initialised clusters by splitting each value ci into two close data points according to the rule : c + i = ci + ffl c?i = ci ? ffl where ffl is a small splitting parameter and i varies from 1 to the current number of of the initialised clusters -LRB- which is 1 in the first case -RRB- ; 3 . Calculate the centroids of the new clusters . 4 . Repeat step 2 and 3 until the desired number of clusters is achieved . One problem with the binary splitting technique is that it requires there be at least 10 or 20 data points in each cluster -LRB- Buzo , 1980 -RRB- , which makes it impractical for applications that use fewer data points in each cluster . Another problem is the \ empty cell '' problem , which occurs when a new centroid is assigned to an empty cluster . This leads to an inefficient use of the cluster centres . 2 -LRB- Huang , 1993 -RRB- proposed Direct-Search Binary Splitting -LRB- DSBS -RRB- method , using the Principle Component Analysis -LRB- PCA -RRB- technique . However , it has been pointed out that there is no point in carrying on the PCA if the data points are approximately uncorrelated -LRB- Chatfield , 1980 , p. 57 -RRB- . Another initialisation method is the Simple Cluster-Seeking -LRB- SCS -RRB- method -LRB- Tou , 1974 -RRB- . It should be noted that this method was designed to be a clustering method rather than an initialisation method . However , it can be used to provide initialisation for other cluster methods -LRB- see -LRB- Gersho , 1992 , p. 362 -RRB- -RRB- . The SCS has the following steps : Initialise the first cluster centre with the value of the first data point . Next compute the distance between this -LRB- first -RRB- cluster centre and the next data point . If it is less than some threshold , continue , else add the new data point to the cluster centres . With each new point , find the nearest cluster centre . If the resulting distance is not within the threshold , add the data point to the cluster centres . Continue in this fashion until the desired number of clusters is achieved . If the number of cluster is not achieved , the threshold value must be reduced and the process is repeated . The above method depends on the order in which the data points are considered and the threshold settings . -LRB- Babu , 1993 -RRB- used Genetic Algorithms -LRB- GAs -RRB- to initialise the clusters . GAs start with one population of a fixed-size of candidate solutions -LRB- strings -RRB- . During each iteration step , called a generation , the solutions of the current population are evaluated , and , on the basis of those evaluations , a new population of candidate solutions is formed . 3 Solutions in a population are evaluated based on their objective -LRB- fitness -RRB- value in order to evolve a new population . From the current population , the next population is generated using three probabilistic operators : reproduction , crossover and mutation operators . These operators play an important role in allowing solutions with high fitness to contribute in the next generation . To initialise the clusters , each solution in the population is mapped into a value . The k-means algorithm is then run using each of these values as an initial cluster value and the value with the least within-group sum of square distance is chosen as the final value to initialise the cluster . This process is continued until a maximum number of generations is reached . However , the complexity of this technique is very high , since the k-means is run as many times as the number of the values -LRB- solutions -RRB- in each population , which is an expensive process . Most recently , -LRB- Katsavounidis , 1994 -RRB- proposed a method -LRB- it will be termed KKZ -RRB- which gave better results than the BS technique . The method works in the following way : 1 . Initialise the first cluster with the data point at x , say , with maximum norm . 2 . Initialise the second cluster with the data point which is farthest from x ; 3 . Compute the minimum distances between each of the remaining data points and all current initial cluster centre values , find the data point with the largest value of of these minimum distances and assign it as 4 the next cluster initial value ; 4 . Repeat step 3 until the desired number of clusters is initialised . In the following sections , two methods are presented for cluster initialisation . These methods require the division of the data space into a number of cells . Both methods initialise the clusters according to the density of data points , in method-1 so that the density of cluster centroids is proportional to density of data points ; method-2 provides an optimal solution under simplifying assumptions . Both methods are tested on different data sets and compared with the SCS and KKZ methods . The results show that , for the data sets employed , the new methods lead to better solutions . 2 The New Methods In this section , two new methods for cluster initialisation are presented . The methods require the data space to be divided into a number of cells . The initialisation is done once and there is no need to try different initialisation sets . Further , there is no lower bound on the number of points per cluster for these methods to give good results . The idea of the methods is to initialise cluster centres according to the data distribution at a \ macro '' level and to rely on clustering to yield an improved solution at the detailed level . 5 2.1 Method-1 This method distributes cluster centres straightforwardly , according to the density of the data points . The algorithm is as follows : Assume that the space is divided into equally sized cells ; let N be the number of data points and C the number of desired clusters . let NCi be the number of data points in cell , i. let M be the number of cells . 1 . For each cell compute the integer part of KCi = NCi ? C = N ; 2 . Select randomly KCi data points in cell i as initial cluster centres for data points in cell i . 3 . If -LRB- PMi = 1 KCi = C -RRB- STOP . 4 . If -LRB- PMi = 1 KCi < C -RRB- let DEF = C ? PMi = 1 KCi ; if DEF is large , say > 10 % of C , let M = M ? -LRB- pM ? 1 -RRB- 2 ; go to step 1 ; else select randomly DEF data points ; STOP . 2.2 Method-2 The second method recognises that , if the objective is to minimise the sum of squares of distances from data points to cluster centroids , then the optimal distribution of cluster centroids will balance few large distances in sparse regions with many small distances in dense regions . We first suppose that the total volume , V containing the data points is divided into two volumes of size v and -LRB- V ? v -RRB- . 6 Suppose p data points are randomly distributed throughout volume v , and -LRB- N?p -RRB- data points are randomly distributed throughout volume -LRB- V ? v -RRB- , where N is the number of data points in volume V . We wish to place n cluster centres in volume v , -LRB- C ? n -RRB- in area -LRB- V ? v -RRB- , such that S = X -LRB- d2 -RRB- is minimised ; where d is the distance from each data point to the nearest cluster centre , and the sum is over all the data points . Now , it can be shown that , in a k-dimensional space , if the data points are randomly distributed , and if the cluster centres are placed on a regular grid then , ignoring boundary effects , the expected square of the distance between any data point and its nearest cluster centre within volume v is proportional to -LRB- v = n -RRB- 2 = k. Similarly , the expectation for the square of the distance between data point and nearest cluster centre in volume -LRB- V ? v -RRB- is -LSB- -LRB- V ? v -RRB- = -LRB- C ? n -RRB- -RSB- 2 = k . So our best estimate for S is : S = P ? -LRB- p ? -LRB- v = n -RRB- 2 = k + -LRB- -LRB- N ? p -RRB- ? -LSB- -LRB- V ? v -RRB- = -LRB- C ? n -RRB- -RSB- 2 = k where P is some proportionality constant . Taking the first derivative of S with respect to n shows that S will be minimised when : n -LRB- 2 + k -RRB- = -LRB- pk ? v2 -RRB- = -LRB- C ? n -RRB- -LRB- 2 + k -RRB- =-LRB- -LRB- N ? p -RRB- k ? -LRB- V ? v -RRB- 2 -RRB- 7 This is easily solved numerically for n which should be rounded to the nearest integer . For the two-dimensional case it reduces to the quadratic equation : ? p ? v = n2 + -LRB- N ? p -RRB- ? -LRB- V ? v -RRB- = -LRB- C ? n -RRB- 2 = which can be solved in the usual way . This second method can be modified by repeatedly subdividing volumes v and -LRB- V ? v -RRB- etc. into two , until some lower bound on the number of clusters in each sub area is reached . The method would be expected to work well if some intelligence were used in selecting how the division were made -LRB- ie corresponding to sparse and dense regions -RRB- ; note that v and V-v need not be of equal size . However for our experiments , in order to present a fully-automated method , each binary division results in two equally-sized volumes . The experiments have also been restricted to the two-dimensional case . 3 Experimental Results As discussed in -LRB- Babu , 1993 ; Venkateswarlu , 1992 -RRB- , there is no general proof of convergence for the k-means clustering method . However , there exist some techniques for measuring clustering performance . One of these techniques is the use of the sum of square-distances between data points and their cluster centre . That is E = NX i = 1 d2i ; c -LRB- i -RRB- 8 clusters SCS KKZ Method-1 Method-2 600 1.818 .822 .809 .777 700 1.535 .700 .669 .627 800 1.125 .605 .570 .510 900 1.003 .434 .437 .326 1000 0.786 .308 .415 .237 Table 1 : Final Sum of Square-Distances for the generated Data where N is the number of data points and di ; c -LRB- i -RRB- is the distance from data point i , to the cluster centre to which it is assigned . This technique has been suggested in -LRB- Babu , 1993 ; Gersho , 1992 ; Wan , 1988 -RRB- The technique allows two solutions to be compared for a given data set , the smaller the value of E , the better the solution . The new methods have been applied to two sets of data points to compute different sets of clusters . The first data set -LRB- which contained 10,000 data points -RRB- was artificially generated to simulate nine areas of dense population against a sparsely populated background while in the second set -LRB- which contained 50,000 data points -RRB- the data points correspond to positions of water hydrants in an area including the city of Leeds . In Tables 1 and 2 are presented final results -LRB- after applying the k-means algorithm -RRB- for all methods studied using the same stopping criteria and the best solution from different threshold values for the SCS method . Table 1 shows that for the generated data method-2 performed the best and the SCS performed the 9 clusters SCS KKZ Method-1 Method-2 Init Fin Init Fin Init Fin Init Fin 600 1890.2 21.296 3.592 2.222 4.231 2.149 3.878 2.111 700 1884.4 20.265 3.023 1.877 3.714 1.866 3.258 1.786 800 1876.9 18.231 2.539 1.611 3.201 1.576 2.743 1.532 900 1789.4 16.134 2.164 1.395 3.101 1.415 2.493 1.345 1000 1663.1 14.976 1.907 1.249 2.331 1.226 2.212 1.221 Table 2 : Initial & Final Sum of Square-Distances for the Hydrant Data worst , while method-1 outperformed the KKZ in most cases . Table 2 shows similar results for the hydrant data where the new methods outperformed the other two , although the KKZ method obtained better starting initial values . Also in Table 2 is shown the sum of square-distances resulting from the initialisation . Figure 1 shows the performance of the new methods . Although our experiments did not show a significant change on the results by using different numbers of cells , however , the best results were obtained when the number of cells , M is approximately equal C = 2pC , where C is the number of desired clusters . 10 1 2 3 600 700 800 900 1000 Sum of Square-Distances Number of clusters Generated Data : KKZ Method-1 Method-2 Hydrant Data : KKZ Method-1 Method-2 Figure 1 : Final Sum of Square Distances for the Generated & Hydrant data 4 Summary Two methods for cluster initialisation have been presented in this letter . Both methods gave better results than the SCS and ZZK methods and method-2 gave the best results . The SCS method gave the worst results , particularly for larger number of data points . Although the KKZ gave the best initial results , the new methods gave better results after applying to the k-means algorithm . This result is not unexpected as our new initialisation methods do not attempt to optimise at the detailed level . 11 Acknowledgement The authors would like to thank Yorkshire Water for making available the water hydrant data used for this study . References Babu G. and M. Murty -LRB- 1993 -RRB- . A near-optimal initial seed value selection in K-means algorithm using a genetic algorithm . Pattern Recognition Letters 14 , 763 -LCB- 769 . Buzo A. , A. Gray , R. Gray and J. Markel -LRB- 1980 -RRB- . Speech Coding Based Upon Vector Quantization . IEEE trans . Acoust . Speech . Signal Process . ASSP-28 , 562 -LCB- 574 . Chatfield C. and A. Collins -LRB- 1980 -RRB- . Introduction to Multivariate Analysis . Chapman and Hall . Gersho A. and R. Gray -LRB- 1992 -RRB- . Vector Quantization and Signal Compression . CAP . Huang C. and R. Harris -LRB- 1993 -RRB- . A Comparison of Several Vector Quantization Codebook Generation Approaches . IEEE trans . Image Proc . 2 -LRB- 1 -RRB- , 108 -LCB- 112 . Ismail M. and M. Kamel -LRB- 1989 -RRB- . Multidimensional Data Clustering Utilization Hybrid Search Strategies . Pattern Recognition 22 -LRB- 1 -RRB- , 75 -LCB- 89 . Kaufman L. and P. Rousseeuw -LRB- 1990 -RRB- . Finding Groups in Data : An Introduction to Cluster Analysis . John Wiley . Katsavounidis I. , C. Kuo and Z. Zhang -LRB- 1994 -RRB- . A New Initialization Technique for Generalized Lioyd Iteration . IEEE Signal Proc . Letters 1 -LRB- 10 -RRB- , 12 144 -LCB- 146 . Linde Y. , A. Buzo and R. Gray -LRB- 1980 -RRB- . An Algorithm for Vector Quantizer Design . IEEE trans . Comm . 28 -LRB- 1 -RRB- , 84 -LCB- 95 . MacQueen J. -LRB- 1967 -RRB- . Some Methods for Classification and Analysis of Multivariate Observations . Proc . 5th Berkeley Symp . Math . Stat . and Prob , 281 -LCB- 297 . Nasrabadi N. and R. King -LRB- 1988 -RRB- . Image Coding using Vector Quantization : A Review . IEEE trans . Comm . 36 -LRB- 8 -RRB- , 957-970 . Tou J. and R. Gonzales -LRB- 1974 -RRB- . Pattern Recognition Principles . AddisonWesley . Reading MA . Venkateswarlu N. and P. Raju -LRB- 1992 -RRB- . Fast Isodata Clustering Algorithms . Pattern Recognition 25 -LRB- 3 -RRB- , 335 -LCB- 342 . Wan S. , S. Wong and P. Prusinkiewicz -LRB- 1988 -RRB- . An Algorithm for Multidimensional Data Clustering . ACM trans . Math . Soft . 14 -LRB- 2 -RRB- , 153 -LCB- 162 . 13

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O