Hierarchical clustering

From Wikipedia, the free encyclopedia
Jump to: navigation, search


  • Divisive: This is a "top down" approach: all observations start in one cluster, and splits are performed recursively as one moves down the hierarchy.


In the general case, the complexity of agglomerative clustering is O(n^3), which makes them too slow for large data sets. Divisive clustering with an exhaustive search is O(2^n), which is even worse. However, for some special cases, optimal efficient agglomerative methods (of complexity O(n^2)) are known: SLINK[2] for single-linkage and CLINK[3] for complete-linkage clustering.

Cluster dissimilarity[edit]

In order to decide which clusters should be combined (for agglomerative), or where a cluster should be split (for divisive), a measure of dissimilarity between sets of observations is required. In most methods of hierarchical clustering, this is achieved by use of an appropriate metric (a measure of distance between pairs of observations), and a linkage criterion which specifies the dissimilarity of sets as a function of the pairwise distances of observations in the sets.


Further information: Metric (mathematics)

The choice of an appropriate metric will influence the shape of the clusters, as some elements may be close to one another according to one distance and farther away according to another. For example, in a 2-dimensional space, the distance between the point (1,0) and the origin (0,0) is always 1 according to the usual norms, but the distance between the point (1,1) and the origin (0,0) can be 2 under Manhattan distance, \scriptstyle\sqrt{2} under Euclidean distance, or 1 under maximum distance.

Some commonly used metrics for hierarchical clustering are:[4]

Names Formula
Euclidean distance  \|a-b \|_2 = \sqrt{\sum_i (a_i-b_i)^2}
Squared Euclidean distance  \|a-b \|_2^2 = \sum_i (a_i-b_i)^2
Manhattan distance  \|a-b \|_1 = \sum_i |a_i-b_i|
maximum distance  \|a-b \|_\infty = \max_i |a_i-b_i|
Mahalanobis distance  \sqrt{(a-b)^{\top}S^{-1}(a-b)} where S is the Covariance matrix

For text or other non-numeric data, metrics such as the Hamming distance or Levenshtein distance are often used.


Linkage criteria[edit]

The linkage criterion determines the distance between sets of observations as a function of the pairwise distances between observations.

Some commonly used linkage criteria between two sets of observations A and B are:[5][6]

Names Formula
Maximum or complete-linkage clustering  \max \, \{\, d(a,b) : a \in A,\, b \in B \,\}.
Minimum or single-linkage clustering  \min \, \{\, d(a,b) : a \in A,\, b \in B \,\}.
Mean or average linkage clustering, or UPGMA  \frac{1}{|A| |B|} \sum_{a \in A }\sum_{ b \in B} d(a,b).
Centroid linkage clustering, or UPGMC  ||c_s - c_t || where c_s and c_t are the centroids of clusters s and t, respectively.
Minimum energy clustering   \frac {2}{nm}\sum_{i,j=1}^{n,m} \|a_i- b_j\|_2 - \frac {1}{n^2}\sum_{i,j=1}^{n} \|a_i-a_j\|_2 - \frac{1}{m^2}\sum_{i,j=1}^{m} \|b_i-b_j\|_2

where d is the chosen metric. Other linkage criteria include:

  • The sum of all intra-cluster variance.
  • The decrease in variance for the cluster being merged (Ward's criterion).[7]
  • The product of in-degree and out-degree on a k-nearest-neighbor graph (graph degree linkage).[8]
  • The increment of some cluster descriptor (i.e., a quantity defined for measuring the quality of a cluster) after merging two clusters.[9][10][11]


Hierarchical clustering has the distinct advantage that any valid measure of distance can be used. In fact, the observations themselves are not required: all that is used is a matrix of distances.

Example for Agglomerative Clustering[edit]

For example, suppose this data is to be clustered, and the Euclidean distance is the distance metric.

Cutting the tree at a given height will give a partitioning clustering at a selected precision. In this example, cutting after the second row of the dendrogram will yield clusters {a} {b c} {d e} {f}. Cutting after the third row will yield clusters {a} {b c} {d e f}, which is a coarser clustering, with a smaller number but larger clusters.

Raw data

The hierarchical clustering dendrogram would be as such:

Traditional representation


Optionally, one can also construct a distance matrix at this stage, where the number in the i-th row j-th column is the distance between the i-th and j-th elements. Then, as clustering progresses, rows and columns are merged as the clusters are merged and the distances updated. This is a common way to implement this type of clustering, and has the benefit of caching distances between clusters. A simple agglomerative clustering algorithm is described in the single-linkage clustering page; it can easily be adapted to different types of linkage (see below).


  • The mean distance between elements of each cluster (also called average linkage clustering, used e.g. in UPGMA):
 {1 \over {|\mathcal{A}|\cdot|\mathcal{B}|}}\sum_{x \in \mathcal{A}}\sum_{ y \in \mathcal{B}} d(x,y).
  • The sum of all intra-cluster variance.
  • The decrease in variance for the cluster being merged (Ward's method[7])



Open Source Implementations[edit]

Standalone implementations[edit]


  • MATLAB includes hierarchical cluster analysis.
  • SAS includes hierarchical cluster analysis in PROC CLUSTER.[12]
  • Mathematica includes a Hierarchical Clustering Package.
  • NCSS (statistical software) includes hierarchical cluster analysis.
  • SPSS includes hierarchical cluster analysis.
  • Qlucore Omics Explorer includes hierarchical cluster analysis.
  • Stata includes hierarchical cluster analysis.

See also[edit]


  1. ^ Rokach, Lior, and Oded Maimon. "Clustering methods." Data mining and knowledge discovery handbook. Springer US, 2005. 321-352.
  2. ^ "The DISTANCE Procedure: Proximity Measures". SAS/STAT 9.2 Users Guide. SAS Institute. Retrieved 2009-04-26. 
  3. ^ "The CLUSTER Procedure: Clustering Methods". SAS/STAT 9.2 Users Guide. SAS Institute. Retrieved 2009-04-26. 
  4. ^ Székely, G. J. and Rizzo, M. L. (2005) Hierarchical clustering via Joint Between-Within Distances: Extending Ward's Minimum Variance Method, Journal of Classification 22, 151-183.
  5. ^ a b Ward, Joe H. (1963). "Hierarchical Grouping to Optimize an Objective Function". Journal of the American Statistical Association 58 (301): 236–244. doi:10.2307/2282967. JSTOR 2282967. MR 0148188. 
  6. ^ Zhang, et al. "Graph degree linkage: Agglomerative clustering on a directed graph." 12th European Conference on Computer Vision, Florence, Italy, October 7–13, 2012. http://arxiv.org/abs/1208.5092
  7. ^ Zhang, et al. "Agglomerative clustering via maximum incremental path integral." Pattern Recognition (2013).
  8. ^ Zhao, and Tang. "Cyclizing clusters via zeta function of a graph."Advances in Neural Information Processing Systems. 2008.
  9. ^ Ma, et al. "Segmentation of multivariate mixed data via lossy data coding and compression." IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(9) (2007): 1546-1562.
  10. ^ http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#cluster_toc.htm

References and further reading[edit]