On the Use of Relative Validity Indices for Comparing Clustering
  Approaches
Luke W. Yerbury Ricardo J. G. B. Campello G. C. Livingston Jr Mark Goldsworthy Lachlan O'Neil
Relative Validity Indices (RVIs) such as the Silhouette Width Criterion,
Calinski-Harabasz and Davie's Bouldin indices are the most popular tools for
evaluating and optimising applications of clustering. Their ability to rank
collections of candidate partitions has been used to guide the selection of the
number of clusters, and to compare partitions from different clustering
algorithms. Beyond these more conventional tasks, many examples can be found in
the literature where RVIs have been used to compare and select other aspects of
clustering approaches such as data normalisation procedures, data
representation methods, and distance measures. The authors are not aware of any
studies that have attempted to establish the suitability of RVIs for such
comparisons. Moreover, given the impact of these aspects on pairwise
similarities, it is not even immediately obvious how RVIs should be implemented
when comparing these aspects. In this study, we conducted experiments with
seven common RVIs on over 2.7 million clustering partitions for both synthetic
and real-world datasets, encompassing feature-vector and time-series data. Our
findings suggest that RVIs are not well-suited to these unconventional tasks,
and that conclusions drawn from such applications may be misleading. It is
recommended that normalisation procedures, representation methods, and distance
measures instead be selected using external validation on high quality labelled
datasets or carefully designed outcome-oriented objective criteria, both of
which should be informed by relevant domain knowledge and clustering aims.