A Survey on Internal Validity Measure for Cluster Validation
---
paper_title: Automatic subspace clustering of high dimensional data for data mining applications

paper_content:

Data mining applications place special requirements on clustering algorithms including: the ability to find clusters embedded in subspaces of high dimensional data, scalability, end-user comprehensibility of the results, non-presumption of any canonical data distribution, and insensitivity to the order of input records. We present CLIQUE, a clustering algorithm that satisfies each of these requirements. CLIQUE identifies dense clusters in subspaces of maximum dimensionality. It generates cluster descriptions in the form of DNF expressions that are minimized for ease of comprehension. It produces identical results irrespective of the order in which input records are presented and does not presume any specific mathematical form for data distribution. Through experiments, we show that CLIQUE efficiently finds accurate cluster in large high dimensional datasets.
---
paper_title: Performance Evaluation of Some Clustering Algorithms and Validity Indices

paper_content:

In this article, we evaluate the performance of three clustering algorithms, hard K-Means, single linkage, and a simulated annealing (SA) based technique, in conjunction with four cluster validity indices, namely Davies-Bouldin index, Dunn's index, Calinski-Harabasz index, and a recently developed index I. Based on a relation between the index I and the Dunn's index, a lower bound of the value of the former is theoretically estimated in order to get unique hard K-partition when the data set has distinct substructures. The effectiveness of the different validity indices and clustering methods in automatically evolving the appropriate number of clusters is demonstrated experimentally for both artificial and real-life data sets with the number of clusters varying from two to ten. Once the appropriate number of clusters is determined, the SA-based clustering technique is used for proper partitioning of the data into the said number of clusters.
---
paper_title: A dendrite method for cluster analysis

paper_content:

A method for identifying clusters of points in a multidimensional Euclidean space is described and its application to taxonomy considered. It reconciles, in a sense, two different approaches to the investigation of the spatial relationships between the points, viz., the agglomerative and the divisive methods. A graph, the shortest dendrite of Florek etal. (1951a), is constructed on a nearest neighbour basis and then divided into clusters by applying the criterion of minimum within cluster sum of squares. This procedure ensures an effective reduction of the number of possible splits. The method may be applied to a dichotomous division, but is perfectly suitable also for a global division into any number of clusters. An informal indicator of the "best number" of clusters is suggested. It is a"variance ratio criterion" giving some insight into the structure of the points. The method is illustrated by three examples, one of which is original. The results obtained by the dendrite method are compared with those...
---
paper_title: A validity measure for fuzzy clustering

paper_content:

The authors present a fuzzy validity criterion based on a validity function which identifies compact and separate fuzzy c-partitions without assumptions as to the number of substructures inherent in the data. This function depends on the data set, geometric distance measure, distance between cluster centroids and more importantly on the fuzzy partition generated by any fuzzy algorithm used. The function is mathematically justified via its relationship to a well-defined hard clustering validity function, the separation index for which the condition of uniqueness has already been established. The performance of this validity function compares favorably to that of several others. The application of this validity function to color image segmentation in a computer color vision system for recognition of IC wafer defects which are otherwise impossible to detect using gray-scale image processing is discussed. >
---
paper_title: Locally adaptive metrics for clustering high dimensional data

paper_content:

Clustering suffers from the curse of dimensionality, and similarity functions that use all input features with equal relevance may not be effective. We introduce an algorithm that discovers clusters in subspaces spanned by different combinations of dimensions via local weightings of features. This approach avoids the risk of loss of information encountered in global dimensionality reduction techniques, and does not assume any data distribution model. Our method associates to each cluster a weight vector, whose values capture the relevance of features within the corresponding cluster. We experimentally demonstrate the gain in perfomance our method achieves with respect to competitive methods, using both synthetic and real datasets. In particular, our results show the feasibility of the proposed technique to perform simultaneous clustering of genes and conditions in gene expression data, and clustering of very high-dimensional data such as text data.
---
paper_title: Evolution-Based Tabu Search Approach to Automatic Clustering

paper_content:

Traditional clustering algorithms (e.g., the K-means algorithm and its variants) are used only for a fixed number of clusters. However, in many clustering applications, the actual number of clusters is unknown beforehand. The general solution to this type of a clustering problem is that one selects or defines a cluster validity index and performs a traditional clustering algorithm for all possible numbers of clusters in sequence to find the clustering with the best cluster validity. This is tedious and time-consuming work. To easily and effectively determine the optimal number of clusters and, at the same time, construct the clusters with good validity, we propose a framework of automatic clustering algorithms (called ETSAs) that do not require users to give each possible value of required parameters (including the number of clusters). ETSAs treat the number of clusters as a variable, and evolve it to an optimal number. Through experiments conducted on nine test data sets, we compared the ETSA with five traditional clustering algorithms. We demonstrate the superiority of the ETSA in finding the correct number of clusters while constructing clusters with good validity.
---

Title: A Survey on Internal Validity Measure for Cluster Validation
Section 1: Introduction
Description 1: Introduce the concepts of data exploration, data clustering, cluster validity, and the objectives of the paper.

Section 2: Defining an Maximization Objective Function
Description 2: Define the objective function for cluster validity indices, including parameters like compactness and separability.

Section 3: Vital Issues to be considered
Description 3: Highlight critical issues in cluster validation, focusing on the impact of outliers and the importance of excluding noisy data.

Section 4: Cluster Validity Index Assessment
Description 4: Discuss the evaluation parameters of various cluster validity indices and their applicability to different application domains.

Section 5: Intricacy
Description 5: Explore the computational complexity related to estimating the parameters of cluster validity indices.

Section 6: Clustering Validity Assessment: Finding the Optimal Partitioning of a Data set
Description 6: Examine methodologies for determining the optimal partitioning of a data set using cluster validity indices.

Section 7: Performance Evaluation of Some Clustering Algorithms and Validity Indices
Description 7: Evaluate the performance of different clustering algorithms and validity indices, focusing on data sets with distinct sub structures.

Section 8: New indices for cluster validity assessment
Description 8: Introduce new cluster validity indices and discuss their design principles and limitations.

Section 9: A new cluster validity index for prototype based clustering algorithms based on inter-and intra-cluster density
Description 9: Present a validity index for prototype-based clustering algorithms that evaluates inter- and intra-cluster densities.

Section 10: An Evolutionary Cluster Validation Index
Description 10: Discuss a new evolutionary method for cluster validation that uses genetic programming to optimize the number of clusters.

Section 11: A Validity Index based on Connectivity
Description 11: Introduce a connectivity-based validity index designed for data sets with well-separated clusters of various shapes and sizes.

Section 12: Data
Description 12: Address the challenges in clustering high-dimensional data and propose a new cluster validity index for high-dimensional spaces.

Section 13: Concluding Remarks
Description 13: Summarize the review of cluster validity indices, emphasizing the limitations of current metrics and suggesting future research directions.