Image Segmentation Evaluation: A Survey of Unsupervised Methods
---
paper_title: Empirical Evaluation Techniques In Computer Vision

paper_content:

From the Publisher: ::: Empirical Evaluation Techniques in Computer Vision presents methods that allow comparative assessment of algorithms and the accompanying benefits: places computer vision on solid experimental and scientific grounds, assists the development of engineering solutions to practical problems, allows accurate assessments of computer vision research, provides convincing evidence that computer vision research results in practical solutions. The chapters in this volume cover the three main paradigms for evaluating computer vision algorithms. The paradigms are: (1) evaluations that are independently administered, (2) evaluation of a set of algorithms by one research group, and (3) evaluation methods that feature ground truthing procedures as a major component.
---
paper_title: Performance measures for video object segmentation and tracking

paper_content:

We propose measures to evaluate quantitatively the performance of video object segmentation and tracking methods without ground-truth (GT) segmentation maps. The proposed measures are based on spatial differences of color and motion along the boundary of the estimated video object plane and temporal differences between the color histogram of the current object plane and its predecessors. They can be used to localize (spatially and/or temporally) regions where segmentation results are good or bad; and/or they can be combined to yield a single numerical measure to indicate the goodness of the boundary segmentation and tracking results over a sequence. The validity of the proposed performance measures without GT have been demonstrated by canonical correlation analysis with another set of measures with GT on a set of sequences (where GT information is available). Experimental results are presented to evaluate the segmentation maps obtained from various sequences using different segmentation approaches.
---
paper_title: Image segmentation based on merging of sub-optimal

paper_content:

In this paper a heuristic segmentation algorithm is presented based on the oversegmentation of an image. The method uses a set of different segmentations of the image produced previously by standard techniques. These segmentations are combined to create the oversegmented image. They can be performed using different techniques or even the same technique with different initial conditions. Based on this oversegmentation a new method of region merging is proposed. The merging process is guided using only information about the behavior of each pixel in the input segmentations. Therefore, the results are an adequate combination of the features of these segmentations, allowing to mask the negative particularities of individual segmentations. In this work, the quality of the proposal is analyzed with both artificial and real images using a evaluation function as case of study. The results show that our algorithm produces high quality global segmentations from a set of low quality segmentations with reduced execution times.
---
paper_title: Comparison of edge detection algorithms using a structure from motion task

paper_content:

This paper presents an evaluation of edge detector performance. We use the task of structure from motion (SFM) as a "black box" through which to evaluate the performance of edge detection algorithms. Edge detector goodness is measured by how accurately the SFM could recover the known structure and motion from the edge detection of the image sequences. We use a variety of real image sequences with ground truth to evaluate eight different edge detectors from the literature. Our results suggest that ratings of edge detector performance based on pixel-level metrics and on the SFM are well correlated and that detectors such as the Canny detector and Heitger detector offer the best performance.
---
paper_title: Comparison of Edge Detector Performance through Use in an Object Recognition Task

paper_content:

This paper presents an empirical evaluation methodology for edge detectors. Edge detector performance is measured using a particular edge-based object recognition algorithm as a “higher-level” task. A detector's performance is ranked according to the object recognition performance that it generates. We have used a challenging train and test dataset containing 110 images of jeep-like images. Six edge detectors are compared and results suggest that (1) the SUSAN edge detector performs best and (2) the ranking of various edge detectors is different from that found in other evaluations.
---
paper_title: Towards Perceptually Driven Segmentation Evaluation Metrics

paper_content:

To be reliable, an automatic segmentation evaluation metric has to be validated by subjective tests. In this paper, a formal protocol for subjective tests for segmentation quality assessment is presented. The most common artifacts produced by segmentation algorithms are identified and an extensive analysis of their effects on the perceived quality is performed. A psychophysical experiment was performed to assess the quality of video with segmentation errors. The results show how an objective segmentation evaluation metric can be defined as a function of various error types.
---
paper_title: Comparison of Edge Detector Performance through Use in an Object Recognition Task

paper_content:

This paper presents an empirical evaluation methodology for edge detectors. Edge detector performance is measured using a particular edge-based object recognition algorithm as a “higher-level” task. A detector's performance is ranked according to the object recognition performance that it generates. We have used a challenging train and test dataset containing 110 images of jeep-like images. Six edge detectors are compared and results suggest that (1) the SUSAN edge detector performs best and (2) the ranking of various edge detectors is different from that found in other evaluations.
---
paper_title: Threshold Evaluation Techniques

paper_content:

Threshold selection techniques have been used as a basic tool in image segmentation, but little work has been done on the problem of evaluating a threshold of an image. The problem of threshold evaluation is addressed, and two methods are proposed for measuring the "goodness" of a thresholded image, one based on a busyness criterion and the other based on a discrepancy or error criterion. These evaluation techniques are applied to a set of infrared images and are shown to be useful in facilitating threshold selection. In fact, both methods usually result in similar or identical thresholds which yield good segmentations of the images.
---
paper_title: Toward a generic evaluation of image segmentation

paper_content:

Image segmentation plays a major role in a broad range of applications. Evaluating the adequacy of a segmentation algorithm for a given application is a requisite both to allow the appropriate selection of segmentation algorithms as well as to tune their parameters for optimal performance. However, objective segmentation quality evaluation is far from being a solved problem. In this paper, a generic framework for segmentation evaluation is introduced after a brief review of previous work. A metric based on the distance between segmentation partitions is proposed to overcome some of the limitations of existing approaches. Symmetric and asymmetric distance metric alternatives are presented to meet the specificities of a wide class of applications. Experimental results confirm the potential of the proposed measures.
---
paper_title: A Method for Objective Edge Detection Evaluation and Detector Parameter Selection

paper_content:

Subjective evaluation by human observers is usually used to analyze and select an edge detector parametric setup when real-world images are considered. We propose a statistical objective performance analysis and detector parameter selection, using detection results produced by different detector parameters. Using the correspondence between the different detection results, an estimated best edge map, utilized as an estimated ground truth (EGT), is obtained. This is done using both a receiver operating characteristics (ROC) analysis and a Chi-square test, and considers the trade off between information and noisiness in the detection results. The best edge detector parameter set (PS) is then selected by the same statistical approach, using the EGT. Results are demonstrated for several edge detection techniques, and compared to published subjective evaluation results. The method developed here suggests a general tool to assist in practical implementations of parametric edge detectors where an automatic process is required.
---
paper_title: Image thresholding: Some new techniques

paper_content:

Abstract Some of the existing threshold selection techniques have been critically reviewed. Two algorithms based on a new conditional entropy measure of a partitioned image have been formulated. The approximate minimum error thresholding algorithm of Kittler and Illingworth has been implemented considering the Poisson distribution for the gray level instead of the commonly used normal distribution. Justification in support of the Poisson distribution has also been given. This method is found to be much better both from the point of view of convergence and segmented output. The proposed methods have been applied on a number of images and are found to produce good results. Objective evaluation of the thresholds has been done using divergence, region uniformity, correlation between original image and the segmented image, and second order entropy.
---
paper_title: Threshold Evaluation Techniques

paper_content:

Threshold selection techniques have been used as a basic tool in image segmentation, but little work has been done on the problem of evaluating a threshold of an image. The problem of threshold evaluation is addressed, and two methods are proposed for measuring the "goodness" of a thresholded image, one based on a busyness criterion and the other based on a discrepancy or error criterion. These evaluation techniques are applied to a set of infrared images and are shown to be useful in facilitating threshold selection. In fact, both methods usually result in similar or identical thresholds which yield good segmentations of the images.
---
paper_title: Contextual and non-contextual performance evaluation of edge detectors

paper_content:

Abstract This paper presents two new evaluation methods for edge detectors. First is non-contextual and concerns the evaluation of edge detector performance in terms of detection errors. The second contextual method evaluates the performance of edge detectors in the context of image reconstruction. Both methods study the influence of image characteristics and edge detector properties on detector performance. Five detectors are evaluated and the performance is compared.
---
paper_title: Design of statistical measures for the assessment of image segmentation schemes

paper_content:

Image segmentation is discussed for years in numerous papers, but assessing its quality is mainly dealt with in recent works. Quality assessment is a primary concern for anyone working towards better segmentation tools. It both helps to objectively improve segmentation techniques and to compare performances with respect to other similar algorithms. ::: ::: In this paper we use a statistical framework to propose statistical measures capable to describe the performances of a segmentation scheme. All the measures rely on a ground-truth segmentation map that is supposed to be known and that serves as a reference when qualifying the results of any segmentation tool. We derive the analytical expression of several transition probabilities and show how to calculate them. An important conclusion from our study, often overlooked, is that performances can be content dependent, which means that one should adapt a measure to the content of an image.
---
paper_title: Empirical Evaluation Techniques In Computer Vision

paper_content:

From the Publisher: ::: Empirical Evaluation Techniques in Computer Vision presents methods that allow comparative assessment of algorithms and the accompanying benefits: places computer vision on solid experimental and scientific grounds, assists the development of engineering solutions to practical problems, allows accurate assessments of computer vision research, provides convincing evidence that computer vision research results in practical solutions. The chapters in this volume cover the three main paradigms for evaluating computer vision algorithms. The paradigms are: (1) evaluations that are independently administered, (2) evaluation of a set of algorithms by one research group, and (3) evaluation methods that feature ground truthing procedures as a major component.
---
paper_title: Context dependent edge detection and evaluation

paper_content:

Abstract To simulate the edge perception ability of human eyes and detect scene edges from an image, context information must be employed in the edge detection process. To accomplish the optimal use of context, we introduce an edge detection scheme which uses the context of the whole image. The edge context for each pixel is the set of all row monotonically increasing paths through the pixel. The edge detector assigns a pixel that edge state having highest edge probability among all the paths. Based on the same framework, we developed a general robust evaluator for edge detectors. The scheme, based on local edge coherence, does not require any prior information about the ideal edge image and allows any size of neighborhood with which local edge coherence is evaluated on the basis of continuity, thinness, and positional accuracy. The edge evaluator can be incorporated with a feedback mechanism to automatically adjust edge detection parameters (e.g. edge thresholds), for adaptive detection of edges in real images. Experiments indicate the validity of the edge detector and the general edge evaluator. Upon comparing the performance of the context dependent edge detector with the context free second directional derivative zero-crossing edge operator, we find that the context dependent edge detector is superior.
---
paper_title: Objective evaluation of relative segmentation quality

paper_content:

When working in image and video segmentation, the major objective is to design an algorithm producing the appropriate segmentation results for the particular goals of the application addressed. Therefore, the assessment of the segmentation quality assumes a crucial importance to evaluate the likeliness that the application targets are met. Since no well-established methods for objective segmentation quality evaluation are currently available, this paper's major goal is to propose objective metrics for the evaluation of relative segmentation quality for both individual objects and the overall segmentation partition. The paper presents a methodology for performing objective evaluation of relative segmentation quality, identifies the relevant features to be compared against those of the reference segmentation, and proposes appropriate objective quality metrics. These metrics build on the existing knowledge of segmentation quality evaluation and also on some relevant aspects from the video quality evaluation field.
---
paper_title: Scene-segmentation algorithm development using error measures.

paper_content:

Development of scene-segmentation algorithms has generally been an ad hoc process. This paper presents a systematic technique for developing these algorithms using error-measure minimization. If scene segmentation is regarded as a problem of pixel classification whereby each pixel of a scene is assigned to a particular object class, development of a scene-segmentation algorithm becomes primarily a process of feature selection. In this study, four methods of feature selection were used to develop segmentation techniques for cervical cytology images: (1) random selection, (2) manual selection (best features in the subjective judgment of the investigator), (3) eigenvector selection (ranking features according to the largest contribution to each eigenvector of the feature covariance matrix) and (4) selection using the scene-segmentation error measure A2. Four features were selected by each method from a universe of 35 features consisting of gray level, color, texture and special pixel neighborhood features in 40 cervical cytology images . Evaluation of the results was done with a composite of the scene-segmentation error measure A2, which depends on the percentage of scenes with measurable error, the agreement of pixel class proportions, the agreement of number of objects for each pixel class and the distance of each misclassified pixel to the nearest pixel of the misclassified class. Results indicate that random and eigenvector feature selection were the poorest methods, manual feature selection somewhat better and error-measure feature selection best. The error-measure feature selection method provides a useful, systematic method of developing and evaluating scene-segmentation algorithms.
---
paper_title: Toward a generic evaluation of image segmentation

paper_content:

Image segmentation plays a major role in a broad range of applications. Evaluating the adequacy of a segmentation algorithm for a given application is a requisite both to allow the appropriate selection of segmentation algorithms as well as to tune their parameters for optimal performance. However, objective segmentation quality evaluation is far from being a solved problem. In this paper, a generic framework for segmentation evaluation is introduced after a brief review of previous work. A metric based on the distance between segmentation partitions is proposed to overcome some of the limitations of existing approaches. Symmetric and asymmetric distance metric alternatives are presented to meet the specificities of a wide class of applications. Experimental results confirm the potential of the proposed measures.
---
paper_title: A methodology for quantitative performance evaluation of detection algorithms

paper_content:

We present a methodology for the quantitative performance evaluation of detection algorithms in computer vision. A common method is to generate a variety of input images by varying the image parameters and evaluate the performance of the algorithm, as algorithm parameters vary. Operating curves that relate the probability of misdetection and false alarm are generated for each parameter setting. Such an analysis does not integrate the performance of the numerous operating curves. In this paper, we outline a methodology for summarizing many operating curves into a few performance curves. This methodology is adapted from the human psychophysics literature and is general to any detection algorithm. The central concept is to measure the effect of variables in terms of the equivalent effect of a critical signal variable, which in turn facilitates the determination of the breakdown point of the algorithm. We demonstrate the methodology by comparing the performance of two-line detection algorithms.
---
paper_title: Edge Detector Evaluation Using Empirical ROC Curves

paper_content:

We demonstrate a method for evaluating edge detector performance based on receiver operating characteristic (ROC) curves. Edge detector output is matched against ground truth to count true positive and false positive edge pixels. A detector's parameter settings are trained to give a best ROC curve on one image and then tested on separate images. We compute aggregate ROC curves based on 1 set of 50 object images and another set of 10 aerial images. We analyze the performance of 11 different edge detectors reported in the literature.
---
paper_title: Objective evaluation of segmentation masks in video sequences

paper_content:

In order to avoid the charge of visually evaluating the quality of segmentation masks, the present paper introduces a procedure for evaluating this quality by means of algorithmically computed figures of merit. Assuming the existence of a perfect (reference) mask, generated manually or with a reliable procedure over a test set, these figures of merit take into account visually desirable properties of a segmentation mask in order to provide the user with outputs that best qualify the spatial and temporal accuracy of the segmentation masks. For the sake of easy interpretation, results are presented on a PSNR-like logarithmic scale.
---
paper_title: Objective evaluation of video segmentation quality

paper_content:

Video segmentation assumes a major role in the context of object-based coding and description applications. Evaluating the adequacy of a segmentation result for a given application is a requisite both to allow the appropriate selection of segmentation algorithms as well as to adjust their parameters for optimal performance. Subjective testing, the current practice for the evaluation of video segmentation quality, is an expensive and time-consuming process. Objective segmentation quality evaluation techniques can alternatively be used; however, it is recognized that, so far, much less research effort has been devoted to this subject than to the development of segmentation algorithms. This paper discusses the problem of video segmentation quality evaluation, proposing evaluation methodologies and objective segmentation quality metrics for individual objects as well as for complete segmentation partitions. Both standalone and relative evaluation metrics are developed to cover the cases for which a reference segmentation is missing or available for comparison.
---
paper_title: Comparison of Edge Detector Performance through Use in an Object Recognition Task

paper_content:

This paper presents an empirical evaluation methodology for edge detectors. Edge detector performance is measured using a particular edge-based object recognition algorithm as a “higher-level” task. A detector's performance is ranked according to the object recognition performance that it generates. We have used a challenging train and test dataset containing 110 images of jeep-like images. Six edge detectors are compared and results suggest that (1) the SUSAN edge detector performs best and (2) the ranking of various edge detectors is different from that found in other evaluations.
---
paper_title: A comparative study of supervised evaluation criteria for image segmentation

paper_content:

This paper presents a comparative study of five supervised evaluation criteria for image segmentation. The different criteria have been tested on a selection of hundred images extracted from the ⓒ Corel database for which manual segmentation results provided by experts are available. Nine segmentation algorithms have been considered, most of which are based on threshold selection. In order to compare the behavior of the different criteria towards over- and undersegmentation, three thresholds have been taken into account, for each selected image, to simulate the various situations. Experimental results permit to reveal the advantages and limitations of the studied criteria.
---
paper_title: Validation of the interleaved pyramid for the segmentation of 3D vector images

paper_content:

Abstract A multiresolution pyramid with double scale space sampling (compared to the Burt & Hong scheme) for the segmentation of 3D images, of which the elements are multiple valued, is described. Evaluation is carried out by quality constrained cost analysis (QCCA).
---
paper_title: Scalable discrepancy measures for segmentation evaluation

paper_content:

We propose a set of scalable discrepancy measures that may be applied for segmentation evaluation when a reference is known. The proposed measures take into account under and over detected points within an adjustable area. They give the intensity of the discrepancy and its relative position. Furthermore a scale parameter allows the accuracy of the measures to be adjusted.
---
paper_title: A Measure for Objective Evaluation of Image Segmentation Algorithms

paper_content:

Despite significant advances in image segmentation techniques, evaluation of these techniques thus far has been largely subjective. Typically, the effectiveness of a new algorithm is demonstrated only by the presentation of a few segmented images and is otherwise left to subjective evaluation by the reader. Little effort has been spent on the design of perceptually correct measures to compare an automatic segmentation of an image to a set of hand-segmented examples of the same image. This paper demonstrates how a modification of the Rand index, the Normalized Probabilistic Rand (NPR) index, meets the requirements of largescale performance evaluation of image segmentation. We show that the measure has a clear probabilistic interpretation as the maximum likelihood estimator of an underlying Gibbs model, can be correctly normalized to account for the inherent similarity in a set of ground truth images, and can be computed efficiently for large datasets. Results are presented on images from the publicly available Berkeley Segmentation dataset.
---
paper_title: Quantitative evaluation of a novel image segmentation algorithm

paper_content:

We present a quantitative evaluation of SE-MinCut, a novel segmentation algorithm based on spectral embedding and minimum cut. We use human segmentations from the Berkeley segmentation database as ground truth and propose suitable measures to evaluate segmentation quality. With these measures we generate precision/recall curves for SE-MinCut and three of the leading segmentation algorithms: mean-shift, normalized Cuts, and the local variation algorithm. These curves characterize the performance of each algorithm over a range of input parameters. We compare the precision/recall curves for the four algorithms and show segmented images that support the conclusions obtained from the quantitative evaluation.
---
paper_title: Three-dimensional image segmentation using a split, merge and group approach

paper_content:

A 3-D segmentation algorithm is presented, based on a split, merge and group approach. It uses a mixed (oct/quad)tree implementation. A number of homogeneity criteria is discussed and evaluated. An example shows the segmentation of mythramycin stained cell nuclei.
---
paper_title: Image-Segmentation Evaluation From the Perspective of Salient Object Extraction

paper_content:

Image segmentation and its performance evaluation are very difficult but important problems in computer vision. A major challenge in segmentation evaluation comes from the fundamental conflict between generality and objectivity: For general-purpose segmentation, the ground truth and segmentation accuracy may not be well defined, while embedding the evaluation in a specific application, the evaluation results may not be extended to other applications. We present in this paper a new benchmark for evaluating image segmentation. Specifically, we formulate image segmentation as identifying the single most perceptually salient structure from an image. We collect a large variety of test images that conforms to this specific formulation, construct unambiguous ground truth for each image, and define a reliable way to measure the segmentation accuracy. We then present two special strategies to further address two important issues: (a) the most salient structures in some real images may not be unique or unambiguously defined, and (b) many available image-segmentation methods are not developed to directly extract a single salient structure. Finally, we apply this benchmark to evaluate and compare the performance of several state-of-the-art image-segmentation methods, including the normalized-cut method, the level-set method, the efficient graph-based method, the mean-shift method, and the ratio-contour method.
---
paper_title: Still Image Objective Segmentation Evaluation using Ground Truth

paper_content:

In this paper, an objective segmentation evaluation metric suitable for the evaluation of still image segmentation results is proposed. The proposed metric is based on the spatial accuracy approach, originally proposed for the evaluation of foreground/backgroung segmentation masks generated from video sequences. This approach is extended to still image segmentation evaluation, where both the estimated segmentation masks and the ground truth mask typically contain multiple regions. The proposed method takes into account, using a single metric, not only the accuracy of the boundary localization of the created segments but also the under-segmentation and over-segmentation effects, which can hinder the performance of any segmentation algorithm and decrease the usability of the segmentation results in content-based applications. Several experiments have shown the potential of this approach.
---
paper_title: Objective and quantitative segmentation evaluation and comparison

paper_content:

Abstract A general framework for segmentation evaluation is introduced after a brief review of previous work. The accuracy of object feature measurement is proposed as a criterion for judging the quality of segmentation results and assessing the performance of applied algorithms. This goal-oriented approach has been shown useful for an objective and quantitative study of segmentation techniques.
---
paper_title: Evaluating Image Segmentation Algorithms Using the Pareto Front

paper_content:

Image segmentation is the first stage of processing in many practical computer vision systems. While development of particular segmentation algorithms has attracted considerable research interest, relatively little work has been published on the subject of their evaluation. In this paper we propose the use of the Pareto front to allow evaluation and comparison of image segmentation algorithms in multi-dimensional fitness spaces, in a manner somewhat analogous to the use of receiver operating characteristic curves in binary classification problems. The principle advantage of this approach is that it avoids the need to aggregate metrics capturing multiple objectives into a single metric, and thus allows trade-offs between multiple aspects of algorithm behavior to be assessed. This is in contrast to previous approaches which have tended to use a single measure of "goodness", or discrepancy to ground truth data. We define the Pareto front in the context of algorithm evaluation, propose several fitness measures for image segmentation, and use a genetic algorithm for multi-objective optimization to explore the set of algorithms, parameters, and corresponding points in fitness space which lie on the front. Experimental results are presented for six general-purpose image segmentation algorithms, including several which may be considered state-of-the-art.
---
paper_title: Stand-Alone Objective Segmentation Quality Evaluation

paper_content:

AbstractThe identification of objects in video sequences, that is, video segmentation, plays a major role in emerging interactive multimedia services, such as those enabled by the ISO MPEG- ::: and MPEG- ::: standards. In this context, assessing the adequacy of the identified objects to the application targets, that is, evaluating the segmentation quality, assumes a crucial importance. Video segmentation technology has received considerable attention in the literature, with algorithms being proposed to address various types of applications. However, the segmentation quality performance evaluation of those algorithms is often ad hoc, and a well-established solution is not available. In fact, the field of objective segmentation quality evaluation is still maturing; recently, some more efforts have been made, mainly following the emergence of the MPEG object-based coding and description standards. This paper discusses the problem of objective segmentation quality evaluation in its most difficult scenario: standalone evaluation, that is, when a reference segmentation is not available for comparative evaluation. In particular, objective metrics are proposed for the evaluation of standalone segmentation quality for both individual objects and overall segmentation partitions.
---
paper_title: Performance measures for video object segmentation and tracking

paper_content:

We propose measures to evaluate quantitatively the performance of video object segmentation and tracking methods without ground-truth (GT) segmentation maps. The proposed measures are based on spatial differences of color and motion along the boundary of the estimated video object plane and temporal differences between the color histogram of the current object plane and its predecessors. They can be used to localize (spatially and/or temporally) regions where segmentation results are good or bad; and/or they can be combined to yield a single numerical measure to indicate the goodness of the boundary segmentation and tracking results over a sequence. The validity of the proposed performance measures without GT have been demonstrated by canonical correlation analysis with another set of measures with GT on a set of sequences (where GT information is available). Experimental results are presented to evaluate the segmentation maps obtained from various sequences using different segmentation approaches.
---
paper_title: Design of statistical measures for the assessment of image segmentation schemes

paper_content:

Image segmentation is discussed for years in numerous papers, but assessing its quality is mainly dealt with in recent works. Quality assessment is a primary concern for anyone working towards better segmentation tools. It both helps to objectively improve segmentation techniques and to compare performances with respect to other similar algorithms. ::: ::: In this paper we use a statistical framework to propose statistical measures capable to describe the performances of a segmentation scheme. All the measures rely on a ground-truth segmentation map that is supposed to be known and that serves as a reference when qualifying the results of any segmentation tool. We derive the analytical expression of several transition probabilities and show how to calculate them. An important conclusion from our study, often overlooked, is that performances can be content dependent, which means that one should adapt a measure to the content of an image.
---
paper_title: A comparative study of supervised evaluation criteria for image segmentation

paper_content:

This paper presents a comparative study of five supervised evaluation criteria for image segmentation. The different criteria have been tested on a selection of hundred images extracted from the ⓒ Corel database for which manual segmentation results provided by experts are available. Nine segmentation algorithms have been considered, most of which are based on threshold selection. In order to compare the behavior of the different criteria towards over- and undersegmentation, three thresholds have been taken into account, for each selected image, to simulate the various situations. Experimental results permit to reveal the advantages and limitations of the studied criteria.
---
paper_title: Performance measures for video object segmentation and tracking

paper_content:

We propose measures to evaluate quantitatively the performance of video object segmentation and tracking methods without ground-truth (GT) segmentation maps. The proposed measures are based on spatial differences of color and motion along the boundary of the estimated video object plane and temporal differences between the color histogram of the current object plane and its predecessors. They can be used to localize (spatially and/or temporally) regions where segmentation results are good or bad; and/or they can be combined to yield a single numerical measure to indicate the goodness of the boundary segmentation and tracking results over a sequence. The validity of the proposed performance measures without GT have been demonstrated by canonical correlation analysis with another set of measures with GT on a set of sequences (where GT information is available). Experimental results are presented to evaluate the segmentation maps obtained from various sequences using different segmentation approaches.
---
paper_title: Edge flow: A framework of boundary detection and image segmentation

paper_content:

A novel boundary detection scheme based on "edge flow" is proposed in this paper. This scheme utilizes a predictive coding model to identify the direction of change in color and texture at each image location at a given scale, and constructs an edge flow vector. By iteratively propagating the edge flow, the boundaries can be detected at image locations which encounter two opposite directions of flow in the stable state. A user defined image scale is the only significant control parameter that is needed by the algorithm. The scheme facilitates integration of color and texture into a single framework for boundary detection.
---
paper_title: Color image segmentation based on Markov random field clustering for histological image analysis

paper_content:

In order to characterise the virulence factors of different Mycobacterium tuberculosis strains responsible for the tuberculosis disease, the quantification, by cell counting, of immune cell recruitment is necessary. However this task by microscopic observations is very tedious and difficult to reproduce. Hence we propose an automatic counting approach, consisting in color image segmentation to discriminate three regions: cell nuclei, immune cells and background, followed by the extraction of each cell entity. For color segmentation, a Markov random field clustering approach taking simultaneously into account both color and spatial information is chosen. Our technique was successfully applied to several color images of different strains, and an evaluation of the results has been performed, showing the robustness of the method against noise, marker color changes, illumination changes and blurring.
---
paper_title: Color Snakes

paper_content:

A framework for object segmentation in vector-valued images is presented in this paper. The first scheme proposed is based on geometric active contours moving toward the objects to be detected in the vector-valued image. Object boundaries are obtained as geodesics or minimal weighted-distance curves, where the metric is given by a definition of edges in vector-valued data. The curve flow corresponding to the proposed active contours holds formal existence, uniqueness, stability, and correctness results. The scheme automatically handles changes in the deforming curve topology. The technique is applicable, for example, to color and texture images as well as multiscale representations. We then present an extension of these vector active contours, proposing a possible image flow for vector-valued image segmentation. The algorithm is based on moving each one of the image level sets according to the proposed vector active contours. This extension also shows the relation between active contours and a number of partial-differential-equations-based image processing algorithms as anisotropic diffusion and shock filters.
---
paper_title: Adaptive split-and-merge segmentation based on piecewise least-square approximation

paper_content:

The performance of the classic split-and-merge segmentation algorithm is severely hampered by its rigid split-and-merge processes, which are insensitive to the image semantics. The author proposes efficient algorithms and data structures to optimize the split-and-merge processes by piecewise least-square approximation of image intensity functions. This optimization aims at the unification of segment finding and edge detection. The optimized split-and-merge algorithm is shown to be adaptive to the image semantics and, hence, improves the segmentation validity of the previous algorithms. This algorithm also appears to work well on noisy sources. Since the optimization is done within the split-and-merge framework, the better segmentation performance is achieved at the same order of time complexity as the previous algorithms. >
---
paper_title: Fast adaptive k-means clustering: some empirical results

paper_content:

The authors present learning rate schedules for fast adaptive k -means clustering which surpass the standard MacQueen learning rate schedule (J. MacQeen, 1967) in speed and quality of solution by several orders of magnitude for large k . The methods accomplish this by largely overcoming the problems of metastable local minima and nonstationarity of cluster region boundaries which plague the MacQueen approach. The authors use simulation results to compare the clustering performances of four learning rate schedules applied to independently sampled data from a uniform distribution in one and two dimensions
---
paper_title: The Image Processing Handbook

paper_content:

Consistently rated as the best overall introduction to computer-based image processing, The Image Processing Handbook covers two-dimensional (2D) and three-dimensional (3D) imaging techniques, image printing and storage methods, image processing algorithms, image and feature measurement, quantitative image measurement analysis, and more. Incorporating image processing and analysis examples at all scales, from nano- to astro-, this Seventh Edition: Features a greater range of computationally intensive algorithms than previous versions Provides better organization, more quantitative results, and new material on recent developments Includes completely rewritten chapters on 3D imaging and a thoroughly revamped chapter on statistical analysis Contains more than 1700 references to theory, methods, and applications in a wide variety of disciplines Presents 500+ entirely new figures and images, with more than two-thirds appearing in color The Image Processing Handbook, Seventh Edition delivers an accessible and up-to-date treatment of image processing, offering broad coverage and comparison of algorithms, approaches, and outcomes.
---
paper_title: Learning to detect natural image boundaries using local brightness, color, and texture cues

paper_content:

The goal of this work is to accurately detect and localize boundaries in natural scenes using local image measurements. We formulate features that respond to characteristic changes in brightness, color, and texture associated with natural boundaries. In order to combine the information from these features in an optimal way, we train a classifier using human labeled images as ground truth. The output of this classifier provides the posterior probability of a boundary at each image location and orientation. We present precision-recall curves showing that the resulting detector significantly outperforms existing approaches. Our two main results are 1) that cue combination can be performed adequately with a simple linear model and 2) that a proper, explicit treatment of texture is required to detect boundaries in natural images.
---
paper_title: A co-evaluation framework for improving segmentation evaluation

paper_content:

Object segmentation is an important preprocessing step for many target ::: recognition applications. Many segmentation methods have been studied, ::: but there is still no satisfactory effectiveness measure which makes ::: it hard to compare different segmentation methods, or even different ::: parameterizations of a single method. A good segmentation evaluation ::: method not only would enable different approaches to be compared, but ::: could also be integrated within the target recognition system to ::: adaptively select the appropriate granularity of the segmentation ::: which in turn could improve the recognition accuracy. A few ::: stand-alone effectiveness measures have been proposed, but these ::: measures examine different fundamental criteria of the objects, or ::: examine the same criteria in a different fashion, so they usually work ::: well in some cases, but poorly in the others. We propose a em ::: co-evaluation framework, in which different effectiveness measures ::: judge the performance of the segmentation in different ways, and their ::: measures are combined by using a machine learning approach which ::: coalesces the results. Experimental results demonstrate that our ::: method performs better than the existing methods.
---
paper_title: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics

paper_content:

This paper presents a database containing 'ground truth' segmentations produced by humans for images of a wide variety of natural scenes. We define an error measure which quantifies the consistency between segmentations of differing granularities and find that different human segmentations of the same image are highly consistent. Use of this dataset is demonstrated in two applications: (1) evaluating the performance of segmentation algorithms and (2) measuring probability distributions associated with Gestalt grouping factors as well as statistics of image region properties.
---
paper_title: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics

paper_content:

This paper presents a database containing 'ground truth' segmentations produced by humans for images of a wide variety of natural scenes. We define an error measure which quantifies the consistency between segmentations of differing granularities and find that different human segmentations of the same image are highly consistent. Use of this dataset is demonstrated in two applications: (1) evaluating the performance of segmentation algorithms and (2) measuring probability distributions associated with Gestalt grouping factors as well as statistics of image region properties.
---
paper_title: A co-evaluation framework for improving segmentation evaluation

paper_content:

Object segmentation is an important preprocessing step for many target ::: recognition applications. Many segmentation methods have been studied, ::: but there is still no satisfactory effectiveness measure which makes ::: it hard to compare different segmentation methods, or even different ::: parameterizations of a single method. A good segmentation evaluation ::: method not only would enable different approaches to be compared, but ::: could also be integrated within the target recognition system to ::: adaptively select the appropriate granularity of the segmentation ::: which in turn could improve the recognition accuracy. A few ::: stand-alone effectiveness measures have been proposed, but these ::: measures examine different fundamental criteria of the objects, or ::: examine the same criteria in a different fashion, so they usually work ::: well in some cases, but poorly in the others. We propose a em ::: co-evaluation framework, in which different effectiveness measures ::: judge the performance of the segmentation in different ways, and their ::: measures are combined by using a machine learning approach which ::: coalesces the results. Experimental results demonstrate that our ::: method performs better than the existing methods.
---
paper_title: Meta-Evaluation of Image Segmentation Using Machine Learning

paper_content:

Image segmentation is a fundamental step in many computer vision applications. Generally, the choice of a segmentation algorithm, or parameterization of a given algorithm, is selected at the application level and fixed for all images within that application. Our goal is to create a stand-alone method to evaluate segmentation quality. Stand-alone methods have the advantage that they do not require a manually-segmented reference image for comparison, and can therefore be used for real-time evaluation. Current stand-alone evaluation methods often work well for some types of images, but poorly for others. We propose a meta-evaluation method in which any set of base evaluation methods are combined by a machine learning algorithm that coalesces their evaluations based on a learned weighting function, which depends upon the image to be segmented. The training data used by the machine learning algorithm can be labeled by a human, based on similarity to a human-generated reference segmentation, or based upon system-level performance. Experimental results demonstrate that our method performs better than the existing stand-alone segmentation evaluation methods.
---
paper_title: Epitomic analysis of appearance and shape

paper_content:

We present novel simple appearance and shape models that we call epitomes. The epitome of an image is its miniature, condensed version containing the essence of the textural and shape properties of the image. As opposed to previously used simple image models, such as templates or basis functions, the size of the epitome is considerably smaller than the size of the image or object it represents, but the epitome still contains most constitutive elements needed to reconstruct the image. A collection of images often shares an epitome, e.g., when images are a few consecutive frames from a video sequence, or when they are photographs of similar objects. A particular image in a collection is defined by its epitome and a smooth mapping from the epitome to the image pixels. When the epitomic representation is used within a hierarchical generative model, appropriate inference algorithms can be derived to extract the epitome from a single image or a collection of images and at the same time perform various inference tasks, such as image segmentation, motion estimation, object removal and super-resolution.
---

Title: Image Segmentation Evaluation: A Survey of Unsupervised Methods

Section 1: Introduction
Description 1: This section introduces the topic of image segmentation, the importance of segmentation evaluation, and the challenges associated with it. It also discusses the need for unsupervised evaluation methods and the structure of the paper.

Section 2: The Hierarchy of Current Evaluation Methods
Description 2: This section categorizes and describes the different methodologies used for evaluating image segmentation, including subjective evaluation and various types of objective evaluation methods.

Section 3: Subjective Evaluation
Description 3: This section provides an overview of subjective evaluation methods where segmentation results are judged by human evaluators, discussing their advantages and limitations.

Section 4: System-level Evaluation
Description 4: This section describes methods that examine the impact of segmentation on the overall system or application rather than evaluating the segmentation independently.

Section 5: Analytical Methods
Description 5: This section explains analytical methods that evaluate segmentation algorithms based on the properties of the algorithms themselves, such as processing strategy and resource efficiency.

Section 6: Supervised Evaluation Methods
Description 6: This section discusses supervised evaluation methods that compare segmented images against a manually-segmented reference image, outlining various discrepancy measures used in these methods.

Section 7: Unsupervised Evaluation Methods
Description 7: This section elaborates on unsupervised evaluation methods that evaluate segmented images without requiring a reference image, focusing on the criteria and specific methods proposed in the literature.

Section 8: Current Unsupervised Methods
Description 8: This section summarizes specific unsupervised evaluation methods and the criteria they use, detailing the metrics employed in these methods and how they measure intra-region uniformity and inter-region disparity.

Section 9: Experiments
Description 9: This section details several experiments designed to evaluate the performance of various unsupervised segmentation evaluation methods on synthetic and real-world images.

Section 10: Results Analysis
Description 10: This section analyzes the results of the experiments, discussing the biases and trends observed in the performance of different evaluation methods.

Section 11: Evaluation using Machine Learning
Description 11: This section introduces Meta-evaluation techniques that use machine learning to combine the results of multiple evaluation methods, providing improved overall evaluation accuracy.

Section 12: Conclusions and Future Directions
Description 12: This section summarizes the findings of the paper, highlighting the strengths and shortcomings of current unsupervised evaluation methods, and suggests future research directions for improving segmentation evaluation techniques.