A Review of Kernel Methods in Remote Sensing Data Analysis
---
paper_title: Partially Supervised Oil-Slick Detection by SAR Imagery Using Kernel Expansion

paper_content:

Spaceborne synthetic aperture radar (SAR) is well adapted to detect ocean pollution independently from daily or weather conditions. In fact, oil slicks have a specific impact on ocean wave spectra. Initial wave spectra may be characterized by three kinds of waves, namely big, medium, and small, which correspond physically to gravity and gravity-capillary waves. The increase of viscosity, due to the presence of oil damps gravity-capillary waves. This induces not only a damping of the backscattering to the sensor but also a damping of the energy of the wave spectra. Thus, local segmentation of wave spectra may be achieved by the segmentation of a multiscale decomposition of the original SAR image. In this paper, a semisupervised oil-slick detection is proposed by using a kernel-based abnormal detection into the wavelet decomposition of a SAR image. It performs accurate detection with no consideration to signal stationarity nor to the presence of strong backscatters (such as a ship). The algorithm has been applied on ENVISAT Advanced SAR images. It yields accurate segmentation results even for small slicks, with a very limited number of false alarms.
---
paper_title: Effects of sample size in classifier design

paper_content:

The effect of finite sample-size on parameter estimates and their subsequent use in a family of functions are discussed. General and parameter-specific expressions for the expected bias and variance of the functions are derived. These expressions are then applied to the Bhattacharyya distance and the analysis of the linear and quadratic classifiers, providing insight into the relationship between the number of features and the number of training samples. Because of the functional form of the expressions, an empirical approach is presented to enable asymptotic performance to be accurately estimated using a very small number of samples. Results were experimentally verified using artificial data in controlled cases and using real, high-dimensional data. >
---
paper_title: Kernel-based methods for hyperspectral image classification

paper_content:

This paper presents the framework of kernel-based methods in the context of hyperspectral image classification, illustrating from a general viewpoint the main characteristics of different kernel-based approaches and analyzing their properties in the hyperspectral domain. In particular, we assess performance of regularized radial basis function neural networks (Reg-RBFNN), standard support vector machines (SVMs), kernel Fisher discriminant (KFD) analysis, and regularized AdaBoost (Reg-AB). The novelty of this work consists in: 1) introducing Reg-RBFNN and Reg-AB for hyperspectral image classification; 2) comparing kernel-based methods by taking into account the peculiarities of hyperspectral images; and 3) clarifying their theoretical relationships. To these purposes, we focus on the accuracy of methods when working in noisy environments, high input dimension, and limited training sets. In addition, some other important issues are discussed, such as the sparsity of the solutions, the computational burden, and the capability of the methods to provide outputs that can be directly interpreted as probabilities.
---
paper_title: Statistical learning theory

paper_content:

A comprehensive look at learning and generalization theory. The statistical theory of learning and generalization concerns the problem of choosing desired functions on the basis of empirical data. Highly applicable to a variety of computer science and robotics fields, this book offers lucid coverage of the theory as a whole. Presenting a method for determining the necessary and sufficient conditions for consistency of learning process, the author covers function estimates from small data pools, applying these estimations to real-life problems, and much more.
---
paper_title: A Support Vector Domain Description Approach to Supervised Classification of Remote Sensing Images

paper_content:

This paper addresses the problem of supervised classification of remote sensing images in the presence of incomplete (nonexhaustive) training sets. The problem is analyzed according to two different perspectives: 1) description and recognition of a specific land-cover class by using single-class classifiers and 2) solution of multiclass problems with single-class classification techniques. In this framework, we analyze different one-class classifiers and introduce in the remote sensing community the support vector domain description method (SVDD). The SVDD is a kernel-based method that exhibits intrinsic regularization ability and robustness versus low numbers of high-dimensional samples. The SVDD technique is compared with other standard single-class methods both in problems focused on the recognition of a single specific land-cover class and in multiclass problems. For the latter, we properly define an easily scalable multiclass architecture capable to deal with incomplete training data. Experimental results, obtained on different kinds of data (synthetic, hyperspectral, and multisensor images), point out the effectiveness of the SVDD technique and provide important indications for driving the choice of the classification technique and architecture in the presence of incomplete training data.
---
paper_title: Kernel-Based Framework for Multitemporal and Multisource Remote Sensing Data Classification and Change Detection

paper_content:

The multitemporal classification of remote sensing images is a challenging problem, in which the efficient combination of different sources of information (e.g., temporal, contextual, or multisensor) can improve the results. In this paper, we present a general framework based on kernel methods for the integration of heterogeneous sources of information. Using the theoretical principles in this framework, three main contributions are presented. First, a novel family of kernel-based methods for multitemporal classification of remote sensing images is presented. The second contribution is the development of nonlinear kernel classifiers for the well-known difference and ratioing change detection methods by formulating them in an adequate high-dimensional feature space. Finally, the presented methodology allows the integration of contextual information and multisensor images with different levels of nonlinear sophistication. The binary support vector (SV) classifier and the one-class SV domain description classifier are evaluated by using both linear and nonlinear kernel functions. Good performance on synthetic and real multitemporal classification scenarios illustrates the generalization of the framework and the capabilities of the proposed algorithms.
---
paper_title: Remote Sensing Digital Image Analysis: An Introduction

paper_content:

From the Publisher: ::: The book provides the non-specialist with an introduction to quantitative evaluation of satellite and aircraft derived from remotely retrieved data. Each chapter covers the pros and cons of digital remotely sensed data, without detailed mathematical treatment of computer based algorithms, but in a manner conductive to an understanding of their capabilities and limitations. Problems conclude each chapter.
---
paper_title: Automatic recognition of man-made objects in high resolution optical remote sensing images by SVM classification of geometric image features

paper_content:

Abstract With the advent of Earth Observation satellite sensors producing images in the visible wavelengths with resolutions better than 5 m, it is now possible to recognize man-made objects which were not visible at lower resolutions. Because of the size and the increasing quantity of remote sensing images, tools are needed for computer aided interpretation. In this work we present an image processing system for the detection and recognition of man-made objects in high resolution optical remote sensing images. Detection is understood here as finding a small rectangular area in the image containing an object. Recognition is the attribution of a class label. These algorithms are based on learning methods and on an example data base which contains eleven classes of objects. The examples (more that 150 for each class) have been manually extracted from SPOT 5 THR images (2.5 m resolution). In order to build a system which is independent of the type of object to be recognized, we have used a supervised learning approach based on support vector machines. The system learns a generic model for each class of objects by using a geometric characterization of the examples in the data base. The main novelty of this paper is the use of a high number of geometric image features which allows to characterise several classes of objects with different geometric properties using a supervised learning approach. The results show the possibility of discrimination of several classes of objects with classification rates higher than 80%.
---
paper_title: Kernel-based methods for hyperspectral image classification

paper_content:

This paper presents the framework of kernel-based methods in the context of hyperspectral image classification, illustrating from a general viewpoint the main characteristics of different kernel-based approaches and analyzing their properties in the hyperspectral domain. In particular, we assess performance of regularized radial basis function neural networks (Reg-RBFNN), standard support vector machines (SVMs), kernel Fisher discriminant (KFD) analysis, and regularized AdaBoost (Reg-AB). The novelty of this work consists in: 1) introducing Reg-RBFNN and Reg-AB for hyperspectral image classification; 2) comparing kernel-based methods by taking into account the peculiarities of hyperspectral images; and 3) clarifying their theoretical relationships. To these purposes, we focus on the accuracy of methods when working in noisy environments, high input dimension, and limited training sets. In addition, some other important issues are discussed, such as the sparsity of the solutions, the computational burden, and the capability of the methods to provide outputs that can be directly interpreted as probabilities.
---
paper_title: Spectral and Spatial Classification of Hyperspectral Data Using SVMs and Morphological Profiles

paper_content:

Classification of hyperspectral data with high spatial resolution from urban areas is discussed. An approach has been proposed which is based on using several principal components from the hyperspectral data and build morphological profiles. These profiles can be used all together in one extended morphological profile. A shortcoming of the approach is that it is primarily designed for classification of urban structures and it does not fully utilize the spectral information in the data. Similarly, a pixel-wise classification solely based on the spectral content can be performed, but it lacks information on the structure of the features in the image. An extension is proposed in this paper in order to overcome these dual problems. The proposed method is based on the data fusion of the morphological information and the original hyperspectral data: the two vectors of attributes are concatenated. After a reduction of the dimensionality using Decision Boundary Feature Extraction, the final classification is achieved using a Support Vector Machines classifier. The proposed approach is tested in experiments on ROSIS data from urban areas. Significant improvements are achieved in terms of accuracies when compared to results of approaches based on the use of morphological profiles based on PCs only and conventional spectral classification.
---
paper_title: Classification of hyperspectral remote-sensing data with primal SVM for small-sized training dataset problem☆

paper_content:

With recent technological advances in remote sensing, very high-dimensional (hyperspectral) data are available for a better discrimination among different complex land-cover classes having similar spectral signatures. However, this large number of bands makes very complex the task of automatic data analysis. In the real application, it is difficult and expensive for the expert to acquire enough training samples to learn a classifier. This results in a classification problem with small-size training sample set. Recently, a regularization-based algorithm is usually proposed to handle such problem, such as Support Vector Machine (SVM), which usually are implemented in the dual form with Lagrange theory. However, it can be solved directly in primal formulation. In this paper, we introduces an alternative implementation technique for SVM to address the classification problem with small-size training sample set. It has been empirically proven that the effectiveness of the introduced implementation technique which has been evaluated by benchmark datasets.
---
paper_title: Toward intelligent training of supervised image classifications: directing training data acquisition for SVM classification

paper_content:

Conventional approaches to training a supervised image classification aim to fully describe all of the classes spectrally. To achieve a complete description of each class in feature space, a large training set is typically required. It is not, however, always necessary to have training statistics that provide a complete and representative description of the classes, especially if using nonparametric classifiers. For classification by a support vector machine, only the training samples that are support vectors, which lie on part of the edge of the class distribution in feature space, are required; all other training samples provide no contribution to the classification analysis. If regions likely to furnish support vectors can be identified in advance of the classification, it may be possible to intelligently select useful training samples. The ability to target useful training samples may allow accurate classification from small training sets. This potential for intelligent training sample collection was explored for the classification of agricultural crops from multispectral satellite sensor data. With a conventional approach to training, only a quarter of the training samples acquired actually made a positive contribution to the analysis and allowed the crops to be classified to a high accuracy (92.5%). The majority of the training set, therefore, was unnecessary as it made no contribution to the analysis. Using ancillary information on soil type, however, it would be possible to constrain the training sample acquisition process. By limiting training sample acquisition only to regions with a specific soil type, it was possible to use a small training set to classify the data without loss of accuracy. Thus, a small number of intelligently selected training samples may be used to classify a data set as accurately as a larger training set derived in a conventional manner. The results illustrate the potential to direct training data acquisition strategies to target the most useful training samples to allow efficient and accurate image classification.
---
paper_title: Composite kernels for hyperspectral image classification

paper_content:

This letter presents a framework of composite kernel machines for enhanced classification of hyperspectral images. This novel method exploits the properties of Mercer's kernels to construct a family of composite kernels that easily combine spatial and spectral information. This framework of composite kernels demonstrates: 1) enhanced classification accuracy as compared to traditional approaches that take into account the spectral information only: 2) flexibility to balance between the spatial and spectral information in the classifier; and 3) computational efficiency. In addition, the proposed family of kernel classifiers opens a wide field for future developments in which spatial and spectral information can be easily integrated.
---
paper_title: Kernel-Based Framework for Multitemporal and Multisource Remote Sensing Data Classification and Change Detection

paper_content:

The multitemporal classification of remote sensing images is a challenging problem, in which the efficient combination of different sources of information (e.g., temporal, contextual, or multisensor) can improve the results. In this paper, we present a general framework based on kernel methods for the integration of heterogeneous sources of information. Using the theoretical principles in this framework, three main contributions are presented. First, a novel family of kernel-based methods for multitemporal classification of remote sensing images is presented. The second contribution is the development of nonlinear kernel classifiers for the well-known difference and ratioing change detection methods by formulating them in an adequate high-dimensional feature space. Finally, the presented methodology allows the integration of contextual information and multisensor images with different levels of nonlinear sophistication. The binary support vector (SV) classifier and the one-class SV domain description classifier are evaluated by using both linear and nonlinear kernel functions. Good performance on synthetic and real multitemporal classification scenarios illustrates the generalization of the framework and the capabilities of the proposed algorithms.
---
paper_title: Evaluation of Kernels for Multiclass Classification of Hyperspectral Remote Sensing Data

paper_content:

Classification of hyperspectral remote sensing data with support vector machines (SVMs) is investigated. SVMs have been introduced recently in the field of remote sensing image processing. Using the kernel method, SVMs map the data into higher dimensional space to increase the separability and then fit an optimal hyperplane to separate the data. In this paper, two kernels have been considered. The generalization capability of SVMs as well as the ability of SVMs to deal with high dimensional feature spaces have been tested in the situation of very limited training set. SVMs have been tested on real hyperspectral data. The experimental results show that SVMs used with the two kernels are appropriate for remote sensing classification problems.
---
paper_title: New Support Vector Algorithms

paper_content:

We propose a new class of support vector algorithms for regression and classification. In these algorithms, a parameter nu lets one effectively control the number of support vectors. While this can be useful in its own right, the parameterization has the additional benefit of enabling us to eliminate one of the other free parameters of the algorithm: the accuracy parameter epsilon in the regression case, and the regularization constant C in the classification case. We describe the algorithms, give some theoretical results concerning the meaning and the choice of nu, and report experimental results.
---
paper_title: Remote Sensing Digital Image Analysis: An Introduction

paper_content:

From the Publisher: ::: The book provides the non-specialist with an introduction to quantitative evaluation of satellite and aircraft derived from remotely retrieved data. Each chapter covers the pros and cons of digital remotely sensed data, without detailed mathematical treatment of computer based algorithms, but in a manner conductive to an understanding of their capabilities and limitations. Problems conclude each chapter.
---
paper_title: Support vector domain description

paper_content:

This paper shows the use of a data domain description method, inspired by the support vector machine by Vapnik, called the support vector domain description (SVDD). This data description can be used for novelty or outlier detection. A spherically shaped decision boundary around a set of objects is constructed by a set of support vectors describing the sphere boundary. It has the possibility of transforming the data to new feature spaces without much extra computational cost. By using the transformed data, this SVDD can obtain more flexible and more accurate data descriptions. The error of the first kind, the fraction of the training objects which will be rejected, can be estimated immediately from the description without the use of an independent test set, which makes this method data eAcient. The support vector domain description is compared with other outlier detection methods on real data. ” 1999 Elsevier Science B.V. All rights reserved.
---
paper_title: Support Vector Method for Novelty Detection

paper_content:

Suppose you are given some dataset drawn from an underlying probability distribution P and you want to estimate a "simple" subset S of input space such that the probability that a test point drawn from P lies outside of S equals some a priori specified ν between 0 and 1. ::: ::: We propose a method to approach this problem by trying to estimate a function f which is positive on S and negative on the complement. The functional form of f is given by a kernel expansion in terms of a potentially small subset of the training data; it is regularized by controlling the length of the weight vector in an associated feature space. We provide a theoretical analysis of the statistical performance of our algorithm. ::: ::: The algorithm is a natural extension of the support vector algorithm to the case of unlabelled data.
---
paper_title: Support Vector Method for Novelty Detection

paper_content:

Suppose you are given some dataset drawn from an underlying probability distribution P and you want to estimate a "simple" subset S of input space such that the probability that a test point drawn from P lies outside of S equals some a priori specified ν between 0 and 1. ::: ::: We propose a method to approach this problem by trying to estimate a function f which is positive on S and negative on the complement. The functional form of f is given by a kernel expansion in terms of a potentially small subset of the training data; it is regularized by controlling the length of the weight vector in an associated feature space. We provide a theoretical analysis of the statistical performance of our algorithm. ::: ::: The algorithm is a natural extension of the support vector algorithm to the case of unlabelled data.
---
paper_title: Classification of hyperspectral remote-sensing data with primal SVM for small-sized training dataset problem☆

paper_content:

With recent technological advances in remote sensing, very high-dimensional (hyperspectral) data are available for a better discrimination among different complex land-cover classes having similar spectral signatures. However, this large number of bands makes very complex the task of automatic data analysis. In the real application, it is difficult and expensive for the expert to acquire enough training samples to learn a classifier. This results in a classification problem with small-size training sample set. Recently, a regularization-based algorithm is usually proposed to handle such problem, such as Support Vector Machine (SVM), which usually are implemented in the dual form with Lagrange theory. However, it can be solved directly in primal formulation. In this paper, we introduces an alternative implementation technique for SVM to address the classification problem with small-size training sample set. It has been empirically proven that the effectiveness of the introduced implementation technique which has been evaluated by benchmark datasets.
---
paper_title: Assessing the Accuracy of Remotely Sensed Data: Principles and Practices

paper_content:

Introduction Why Accuracy Assessment? Overview Historical Review Aerial Photography Digital Assessments Data Collection Considerations Classification Scheme Statistical Considerations Data Distribution Randomness Spatial Autocorrelation Sample Size Sampling Scheme Sample Unit Reference Data Collection Basic Collection Forms Basic Analysis Techniques Non-Site Specific Assessments Site Specific Assessments Area Estimation/Correction Practicals Impact of Sample Design on Cost Recommendations for Collecting Reference Data ASources of Variation in Reference Data Photo Interpretation vs. Ground Visitation Interpreter Variability Observations vs. Measurements What is Correct? Labeling Map vs. Labeling the Reference Data Qualitative vs. Quantitative Analysis Local vs. Regional vs. Global Assessments Advanced Topics Beyond the Error Matrix Modifying the Error Matrix Fuzzy Set Theory Measuring Variability Complex Data Sets Change Detection Multi-Layer Assessments California Hardwood Rangeland Monitoring Project Case Study Balancing Statistical Validity with Practical Reality Bibliography
---
paper_title: Remote Sensing Digital Image Analysis: An Introduction

paper_content:

From the Publisher: ::: The book provides the non-specialist with an introduction to quantitative evaluation of satellite and aircraft derived from remotely retrieved data. Each chapter covers the pros and cons of digital remotely sensed data, without detailed mathematical treatment of computer based algorithms, but in a manner conductive to an understanding of their capabilities and limitations. Problems conclude each chapter.
---
paper_title: Learning with local and global consistency

paper_content:

We consider the general problem of learning from labeled and unlabeled data, which is often called semi-supervised learning or transductive inference. A principled approach to semi-supervised learning is to design a classifying function which is sufficiently smooth with respect to the intrinsic structure collectively revealed by known labeled and unlabeled points. We present a simple algorithm to obtain such a smooth solution. Our method yields encouraging experimental results on a number of classification problems and demonstrates effective use of unlabeled data.
---
paper_title: Cluster kernels for semi-supervised learning

paper_content:

We propose a framework to incorporate unlabeled data in kernel classifier, based on the idea that two points in the same cluster are more likely to have the same label. This is achieved by modifying the eigenspectrum of the kernel matrix. Experimental results assess the validity of this approach.
---
paper_title: Manifold regularization: A geometric framework for learning from labeled and unlabeled examples

paper_content:

We propose a family of learning algorithms based on a new form of regularization that allows us to exploit the geometry of the marginal distribution. We focus on a semi-supervised framework that incorporates labeled and unlabeled data in a general-purpose learner. Some transductive graph learning algorithms and standard methods including support vector machines and regularized least squares can be obtained as special cases. We use properties of reproducing kernel Hilbert spaces to prove new Representer theorems that provide theoretical basis for the algorithms. As a result (in contrast to purely graph-based approaches) we obtain a natural out-of-sample extension to novel examples and so are able to handle both transductive and truly semi-supervised settings. We present experimental evidence suggesting that our semi-supervised algorithms are able to use unlabeled data effectively. Finally we have a brief discussion of unsupervised and fully supervised learning within our general framework.
---
paper_title: Statistical learning theory

paper_content:

A comprehensive look at learning and generalization theory. The statistical theory of learning and generalization concerns the problem of choosing desired functions on the basis of empirical data. Highly applicable to a variety of computer science and robotics fields, this book offers lucid coverage of the theory as a whole. Presenting a method for determining the necessary and sufficient conditions for consistency of learning process, the author covers function estimates from small data pools, applying these estimations to real-life problems, and much more.
---
paper_title: Semi-Supervised Graph-Based Hyperspectral Image Classification

paper_content:

This paper presents a semi-supervised graph-based method for the classification of hyperspectral images. The method is designed to handle the special characteristics of hyperspectral images, namely, high-input dimension of pixels, low number of labeled samples, and spatial variability of the spectral signature. To alleviate these problems, the method incorporates three ingredients, respectively. First, being a kernel-based method, it combats the curse of dimensionality efficiently. Second, following a semi-supervised approach, it exploits the wealth of unlabeled samples in the image, and naturally gives relative importance to the labeled ones through a graph-based methodology. Finally, it incorporates contextual information through a full family of composite kernels. Noting that the graph method relies on inverting a huge kernel matrix formed by both labeled and unlabeled samples, we originally introduce the Nystro umlm method in the formulation to speed up the classification process. The presented semi-supervised-graph-based method is compared to state-of-the-art support vector machines in the classification of hyperspectral data. The proposed method produces better classification maps, which capture the intrinsic structure collectively revealed by labeled and unlabeled points. Good and stable accuracy is produced in ill-posed classification problems (high dimensional spaces and low number of labeled samples). In addition, the introduction of the composite-kernel framework drastically improves results, and the new fast formulation ranks almost linearly in the computational cost, rather than cubic as in the original method, thus allowing the use of this method in remote-sensing applications.
---
paper_title: An Adaptive Classifier Design for High-Dimensional Data Analysis with a Limited Training Data Set

paper_content:

Proposes a self-learning and self-improving adaptive classifier to mitigate the problem of small training sample size that can severely affect the recognition accuracy of classifiers when the dimensionality of the multispectral data is high. This proposed adaptive classifier utilizes classified samples (referred to as semilabeled samples) in addition to original training samples iteratively. In order to control the influence of semilabeled samples, the proposed method gives full weight to the training samples and reduced weight to semilabeled samples. The authors show that by using additional semilabeled samples that are available without extra cost, the additional class label information may be extracted and utilized to enhance statistics estimation and hence improve the classifier performance, and therefore the Hughes phenomenon (peak phenomenon) may be mitigated. Experimental results show this proposed adaptive classifier can improve the classification accuracy as well as representation of estimated statistics significantly.
---
paper_title: Mean Map Kernel Methods for Semisupervised Cloud Classification

paper_content:

Remote sensing image classification constitutes a challenging problem since very few labeled pixels are typically available from the analyzed scene. In such situations, labeled data extracted from other images modeling similar problems might be used to improve the classification accuracy. However, when training and test samples follow even slightly different distributions, classification is very difficult. This problem is known as sample selection bias. In this paper, we propose a new method to combine labeled and unlabeled pixels to increase classification reliability and accuracy. A semisupervised support vector machine classifier based on the combination of clustering and the mean map kernel is proposed. The method reinforces samples in the same cluster belonging to the same class by combining sample and cluster similarities implicitly in the kernel space. A soft version of the method is also proposed where only the most reliable training samples, in terms of likelihood of the image data distribution, are used. Capabilities of the proposed method are illustrated in a cloud screening application using data from the MEdium Resolution Imaging Spectrometer (MERIS) instrument onboard the European Space Agency ENVISAT satellite. Cloud screening constitutes a clear example of sample selection bias since cloud features change to a great extent depending on the cloud type, thickness, transparency, height, and background. Good results are obtained and show that the method is particularly well suited for situations where the available labeled information does not adequately describe the classes in the test data.
---
paper_title: Semisupervised Image Classification With Laplacian Support Vector Machines

paper_content:

This letter presents a semisupervised method based on kernel machines and graph theory for remote sensing image classification. The support vector machine (SVM) is regularized with the unnormalized graph Laplacian, thus leading to the Laplacian SVM (LapSVM). The method is tested in the challenging problems of urban monitoring and cloud screening, in which an adequate exploitation of the wealth of unlabeled samples is critical. Results obtained using different sensors, and with low number of training samples, demonstrate the potential of the proposed LapSVM for remote sensing image classification.
---
paper_title: A Novel Transductive SVM for Semisupervised Classification of Remote-Sensing Images

paper_content:

This paper introduces a semisupervised classification method that exploits both labeled and unlabeled samples for addressing ill-posed problems with support vector machines (SVMs). The method is based on recent developments in statistical learning theory concerning transductive inference and in particular transductive SVMs (TSVMs). TSVMs exploit specific iterative algorithms which gradually search a reliable separating hyperplane (in the kernel space) with a transductive process that incorporates both labeled and unlabeled samples in the training phase. Based on an analysis of the properties of the TSVMs presented in the literature, a novel modified TSVM classifier designed for addressing ill-posed remote-sensing problems is proposed. In particular, the proposed technique: 1) is based on a novel transductive procedure that exploits a weighting strategy for unlabeled patterns, based on a time-dependent criterion; 2) is able to mitigate the effects of suboptimal model selection (which is unavoidable in the presence of small-size training sets); and 3) can address multiclass cases. Experimental results confirm the effectiveness of the proposed method on a set of ill-posed remote-sensing classification problems representing different operative conditions
---
paper_title: Learning with labeled and unlabeled data

paper_content:

In this paper, on the one hand, we aim to give a review on literature dealing with the problem of supervised learning aided by additional unlabeled data. On the other hand, being a part of the author's first year PhD report, the paper serves as a frame to bundle related work by the author as well as numerous suggestions for potential future work. Therefore, this work contains more speculative and partly subjective material than the reader might expect from a literature review. We give a rigorous definition of the problem and relate it to supervised and unsupervised learning. The crucial role of prior knowledge is put forward, and we discuss the important notion of input-dependent regularization. We postulate a number of baseline methods, being algorithms or algorithmic schemes which can more or less straightforwardly be applied to the problem, without the need for genuinely new concepts. However, some of them might serve as basis for a genuine method. In the literature review, we try to cover the wide variety of (recent) work and to classify this work into meaningful categories. We also mention work done on related problems and suggest some ideas towards synthesis. Finally, we discuss some caveats and tradeoffs of central importance to the problem.
---
paper_title: Kernel-based methods for hyperspectral image classification

paper_content:

This paper presents the framework of kernel-based methods in the context of hyperspectral image classification, illustrating from a general viewpoint the main characteristics of different kernel-based approaches and analyzing their properties in the hyperspectral domain. In particular, we assess performance of regularized radial basis function neural networks (Reg-RBFNN), standard support vector machines (SVMs), kernel Fisher discriminant (KFD) analysis, and regularized AdaBoost (Reg-AB). The novelty of this work consists in: 1) introducing Reg-RBFNN and Reg-AB for hyperspectral image classification; 2) comparing kernel-based methods by taking into account the peculiarities of hyperspectral images; and 3) clarifying their theoretical relationships. To these purposes, we focus on the accuracy of methods when working in noisy environments, high input dimension, and limited training sets. In addition, some other important issues are discussed, such as the sparsity of the solutions, the computational burden, and the capability of the methods to provide outputs that can be directly interpreted as probabilities.
---
paper_title: Regularization Networks and Support Vector Machines

paper_content:

Regularization Networks and Support Vector Machines are techniques for solving certain problems of learning from examples – in particular, the regression problem of approximating a multivariate function from sparse data. Radial Basis Functions, for example, are a special case of both regularization and Support Vector Machines. We review both formulations in the context of Vapnik's theory of statistical learning which provides a general foundation for the learning problem, combining functional analysis and statistics. The emphasis is on regression: classification is treated as a special case.
---
paper_title: Manifold regularization: A geometric framework for learning from labeled and unlabeled examples

paper_content:

We propose a family of learning algorithms based on a new form of regularization that allows us to exploit the geometry of the marginal distribution. We focus on a semi-supervised framework that incorporates labeled and unlabeled data in a general-purpose learner. Some transductive graph learning algorithms and standard methods including support vector machines and regularized least squares can be obtained as special cases. We use properties of reproducing kernel Hilbert spaces to prove new Representer theorems that provide theoretical basis for the algorithms. As a result (in contrast to purely graph-based approaches) we obtain a natural out-of-sample extension to novel examples and so are able to handle both transductive and truly semi-supervised settings. We present experimental evidence suggesting that our semi-supervised algorithms are able to use unlabeled data effectively. Finally we have a brief discussion of unsupervised and fully supervised learning within our general framework.
---
paper_title: Manifold regularization: A geometric framework for learning from labeled and unlabeled examples

paper_content:

We propose a family of learning algorithms based on a new form of regularization that allows us to exploit the geometry of the marginal distribution. We focus on a semi-supervised framework that incorporates labeled and unlabeled data in a general-purpose learner. Some transductive graph learning algorithms and standard methods including support vector machines and regularized least squares can be obtained as special cases. We use properties of reproducing kernel Hilbert spaces to prove new Representer theorems that provide theoretical basis for the algorithms. As a result (in contrast to purely graph-based approaches) we obtain a natural out-of-sample extension to novel examples and so are able to handle both transductive and truly semi-supervised settings. We present experimental evidence suggesting that our semi-supervised algorithms are able to use unlabeled data effectively. Finally we have a brief discussion of unsupervised and fully supervised learning within our general framework.
---
paper_title: Support vector domain description

paper_content:

This paper shows the use of a data domain description method, inspired by the support vector machine by Vapnik, called the support vector domain description (SVDD). This data description can be used for novelty or outlier detection. A spherically shaped decision boundary around a set of objects is constructed by a set of support vectors describing the sphere boundary. It has the possibility of transforming the data to new feature spaces without much extra computational cost. By using the transformed data, this SVDD can obtain more flexible and more accurate data descriptions. The error of the first kind, the fraction of the training objects which will be rejected, can be estimated immediately from the description without the use of an independent test set, which makes this method data eAcient. The support vector domain description is compared with other outlier detection methods on real data. ” 1999 Elsevier Science B.V. All rights reserved.
---
paper_title: Semisupervised Image Classification With Laplacian Support Vector Machines

paper_content:

This letter presents a semisupervised method based on kernel machines and graph theory for remote sensing image classification. The support vector machine (SVM) is regularized with the unnormalized graph Laplacian, thus leading to the Laplacian SVM (LapSVM). The method is tested in the challenging problems of urban monitoring and cloud screening, in which an adequate exploitation of the wealth of unlabeled samples is critical. Results obtained using different sensors, and with low number of training samples, demonstrate the potential of the proposed LapSVM for remote sensing image classification.
---
paper_title: Statistical learning theory

paper_content:

A comprehensive look at learning and generalization theory. The statistical theory of learning and generalization concerns the problem of choosing desired functions on the basis of empirical data. Highly applicable to a variety of computer science and robotics fields, this book offers lucid coverage of the theory as a whole. Presenting a method for determining the necessary and sufficient conditions for consistency of learning process, the author covers function estimates from small data pools, applying these estimations to real-life problems, and much more.
---
paper_title: A Novel Transductive SVM for Semisupervised Classification of Remote-Sensing Images

paper_content:

This paper introduces a semisupervised classification method that exploits both labeled and unlabeled samples for addressing ill-posed problems with support vector machines (SVMs). The method is based on recent developments in statistical learning theory concerning transductive inference and in particular transductive SVMs (TSVMs). TSVMs exploit specific iterative algorithms which gradually search a reliable separating hyperplane (in the kernel space) with a transductive process that incorporates both labeled and unlabeled samples in the training phase. Based on an analysis of the properties of the TSVMs presented in the literature, a novel modified TSVM classifier designed for addressing ill-posed remote-sensing problems is proposed. In particular, the proposed technique: 1) is based on a novel transductive procedure that exploits a weighting strategy for unlabeled patterns, based on a time-dependent criterion; 2) is able to mitigate the effects of suboptimal model selection (which is unavoidable in the presence of small-size training sets); and 3) can address multiclass cases. Experimental results confirm the effectiveness of the proposed method on a set of ill-posed remote-sensing classification problems representing different operative conditions
---
paper_title: Robust support vector regression for biophysical variable estimation from remotely sensed images

paper_content:

This letter introduces the epsiv-Huber loss function in the support vector regression (SVR) formulation for the estimation of biophysical parameters extracted from remotely sensed data. This cost function can handle the different types of noise contained in the dataset. The method is successfully compared to other cost functions in the SVR framework, neural networks and classical bio-optical models for the particular case of the estimation of ocean chlorophyll concentration from satellite remote sensing data. The proposed model provides more accurate, less biased, and improved robust estimation results on the considered case study, especially significant when few in situ measurements are available
---
paper_title: Retrieval of oceanic chlorophyll concentration with relevance vector machines

paper_content:

Abstract In this communication, we evaluate the performance of the relevance vector machine (RVM) for the estimation of biophysical parameters from remote sensing data. For illustration purposes, we focus on the estimation of chlorophyll-a concentrations from remote sensing reflectance just above the ocean surface. A variety of bio-optical algorithms have been developed to relate measurements of ocean radiance to in situ concentrations of phytoplankton pigments, and ultimately most of these algorithms demonstrate the potential of quantifying chlorophyll-a concentrations accurately from multispectral satellite ocean color data. Both satellite-derived data and in situ measurements are subject to high levels of uncertainty. In this context, robust and stable non-linear regression methods that provide inverse models are desirable. Lately, the use of the support vector regression (SVR) has produced good results in inversion problems, improving state-of-the-art neural networks. However, the SVR has some deficiencies, which could be theoretically alleviated by the RVM. In this paper, performance of the RVM is evaluated in terms of accuracy and bias of the estimations, sparseness of the solutions, robustness to low number of training samples, and computational burden. In addition, some theoretical issues are discussed, such as the sensitivity to training parameters setting, kernel selection, and confidence intervals on the predictions. Results suggest that RVMs offer an excellent trade-off between accuracy and sparsity of the solution, and become less sensitive to the selection of the free parameters. A novel formulation of the RVM that incorporates prior knowledge of the problem is presented and successfully tested, providing better results than standard RVM formulations, SVRs, neural networks, and classical bio-optical models for SeaWIFS, such as Morel, CalCOFI and OC2/OC4 models.
---
paper_title: Estimating Biophysical Parameters from Remotely Sensed Imagery with Gaussian Processes

paper_content:

Recently, a new machine learning approach that is based on the Gaussian process (GP) theory has been introduced in the literature. According to this approach, the learning of a machine (regressor or classifier) is formulated in terms of a Bayesian estimation problem, where the parameters of the machine are assumed to be random variables which follow jointly a Gaussian distribution. The purpose of this work is to investigate this approach in the context of the estimation of biophysical parameters. Experimental results obtained on synthetic and real data, which simulate the spectral behavior of the chlorophyll concentration in subsurface waters, are reported and compared with those yielded by the general regression neural network (GRNN) and the epsiv-insensitive support vector regression (SVR) methods.
---
paper_title: Development and application of a neural network based ocean colour algorithm in coastal waters

paper_content:

An algorithm for determining chlorophyll‐a concentrations in shallow, case II waters has been developed and applied to nearly six years of Sea‐viewing Wide Field‐of‐view Sensor (SeaWiFS) data in order to observe the general chlorophyll‐a patterns in a coastal estuarine environment. Due to the fact that the current empirical chlorophyll‐a algorithm (OC4) used to process SeaWiFS data breaks down in coastal waters, a neural network based algorithm was developed. The neural network in the study uses SeaWiFS remote sensing reflectance data paired with in situ chlorophyll‐a data in the Delaware Bay and its adjacent coastal zone (DBAC) from a number of different days and seasons in an effort to overcome the limitations of single day algorithms and simulated dataset algorithms. Although the neural network model (NN) in this study displayed some difficulty representing high chlorophyll‐a values, it showed significant improvement over the OC4 algorithm. The performance parameters of the NN were an r 2 of 0.79, a ro...
---
paper_title: Biophysical Parameter Estimation With a Semisupervised Support Vector Machine

paper_content:

This letter presents two kernel-based methods for semisupervised regression. The methods rely on building a graph or hypergraph Laplacian with both the available labeled and unlabeled data, which is further used to deform the training kernel matrix. The deformed kernel is then used for support vector regression (SVR). Given the high computational burden involved, we present two alternative formulations based on the Nystrom method and the incomplete Cholesky factorization to achieve operational processing times. The semisupervised SVR algorithms are successfully tested in multiplatform leaf area index estimation and oceanic chlorophyll concentration prediction. Experiments are carried out with both multispectral and hyperspectral data, demonstrating good generalization capabilities when a low number of labeled samples are available, which is usually the case in biophysical parameter retrieval.
---
paper_title: Estimating oceanic chlorophyll concentrations with neural networks

paper_content:

A Neural network (NN) algorithm was constructed to estimate oceanic chlorophyll concentrations from Sea-viewing Wide Field-of-view Sensor (SeaWiFS) data.The algorithm was trained and tested using sample data compiled at the SeaWiFS Bio-optical Algorithm Mini-workshop. A NN using the five visible SeaWiFS bands as inputs with ten nodes in a single hidden layer estimates chlorophyll concentrations efficiently and accurately, modelling the nonlinear transfer function between surface chlorophyll concentrations and remotely sensed reflectance data more accurately than traditional regression methods.
---
paper_title: Inversion methods for physically‐based models

paper_content:

Physically‐based models of vegetation reflectance serve as a basis for extracting vegetation variables using directional and spectral data from modern‐borne sensors (e.g., MODIS, MISR, POLDER, SeaWiFS). Although many models have been inverted, only recently have significant efforts been made to provide operational algorithms. These efforts have exposed a need to significantly improve efficient and accurate methods for inverting these physically‐based models. The characteristics of the traditional inversion, table look‐up, neural network and other methods are discussed as well as the major achievements, advantages/disadvantages, and research issues for each method. The traditional inversion methods using repeated model runs are computationally intensive and are not appropriate for operational application on a per pixel basis for regional and global data. Thus, for larger data sets, simplified (reduced number of variables and/or physical processes) physically‐based models are generally used. The table look‐...
---
paper_title: Robust support vector regression for biophysical variable estimation from remotely sensed images

paper_content:

This letter introduces the epsiv-Huber loss function in the support vector regression (SVR) formulation for the estimation of biophysical parameters extracted from remotely sensed data. This cost function can handle the different types of noise contained in the dataset. The method is successfully compared to other cost functions in the SVR framework, neural networks and classical bio-optical models for the particular case of the estimation of ocean chlorophyll concentration from satellite remote sensing data. The proposed model provides more accurate, less biased, and improved robust estimation results on the considered case study, especially significant when few in situ measurements are available
---
paper_title: Retrieval of oceanic chlorophyll concentration using support vector machines

paper_content:

This letter investigates the possibility of using a new universal approximator-support vector machines (SVMs)-as the nonlinear transfer function between oceanic chlorophyll concentration and marine reflectance. The SeaBAM dataset is used to evaluate the proposed approach. Experimental results show that the SVM performs as well as the optimal multilayer perceptron (MLP) and can be a promising alternative to the conventional MLPs for the retrieval of oceanic chlorophyll concentration from marine reflectance.
---
paper_title: Practical Methods of Optimization

paper_content:

Preface Table of Notation Part 1: Unconstrained Optimization Introduction Structure of Methods Newton-like Methods Conjugate Direction Methods Restricted Step Methods Sums of Squares and Nonlinear Equations Part 2: Constrained Optimization Introduction Linear Programming The Theory of Constrained Optimization Quadratic Programming General Linearly Constrained Optimization Nonlinear Programming Other Optimization Problems Non-Smooth Optimization References Subject Index.
---
paper_title: Application of machine-learning techniques toward the creation of a consistent and calibrated global chlorophyll concentration baseline dataset using remotely sensed ocean color data

paper_content:

This paper introduces a machine-learning approach to satellite ocean color sensor cross calibration. The cross-calibration objective is to eliminate incompatibilities among sensor data from different missions and produce merged daily global ocean color coverage. The approach is designed and investigated using data from the Moderate Resolution Imaging Spectroradiometer (MODIS) onboard of the Terra satellite and Sea-viewing Wide Field-of-view Sensor (SeaWiFS). Data from these two sensors show apparent discrepancies originating from differences in sensor design, calibration, processing algorithms, and from the rate of change in the atmosphere and ocean within 1(1/2) h between sensor imaging of the same regions on the ground. The discrepancies have complex, noisy, and often contradictory time and space variabilities. Support vector machines are used to bring MODIS data to the SeaWiFS representation where SeaWiFS data are considered to exemplify a consistent ocean color baseline. Support vector machines are effective in learning and resolving convoluted data relationships between the two sensors given a variety of bio-optical, atmospheric, viewing geometry, and ancillary information. The method works accurately in low chlorophyll waters and shows a potential to eliminate sensor problems, such as scan angle dependencies and seasonal and spatial trends in data. The results illustrate that MODIS and SeaWiFS differences are noisy and highly variable, which makes it difficult to extrapolate the cross-calibration knowledge onto new time and space domains and to define representative global ocean color datasets for support vector machine training.
---
paper_title: Sparse Bayesian Learning and the Relevance Vector Machine

paper_content:

This paper introduces a general Bayesian framework for obtaining sparse solutions to regression and classification tasks utilising models linear in the parameters. Although this framework is fully general, we illustrate our approach with a particular specialisation that we denote the 'relevance vector machine' (RVM), a model of identical functional form to the popular and state-of-the-art 'support vector machine' (SVM). We demonstrate that by exploiting a probabilistic Bayesian learning framework, we can derive accurate prediction models which typically utilise dramatically fewer basis functions than a comparable SVM while offering a number of additional advantages. These include the benefits of probabilistic predictions, automatic estimation of 'nuisance' parameters, and the facility to utilise arbitrary basis functions (e.g. non-'Mercer' kernels). We detail the Bayesian framework and associated learning algorithm for the RVM, and give some illustrative examples of its application along with some comparative benchmarks. We offer some explanation for the exceptional degree of sparsity obtained, and discuss and demonstrate some of the advantageous features, and potential extensions, of Bayesian relevance learning.
---
paper_title: Maximum likelihood from incomplete data via the EM algorithm

paper_content:

Vibratory power unit for vibrating conveyers and screens comprising an asynchronous polyphase motor, at least one pair of associated unbalanced masses disposed on the shaft of said motor, with the first mass of a pair of said unbalanced masses being rigidly fastened to said shaft and with said second mass of said pair being movably arranged relative to said first mass, means for controlling and regulating the conveying rate during conveyer operation by varying the rotational speed of said motor between predetermined minimum and maximum values, said second mass being movably outwardly by centrifugal force against the pressure of spring means, said spring means being prestressed in such a manner that said second mass is, at rotational motor speeds lower than said minimum speed, held in its initial position, and at motor speeds between said lower and upper values in positions which are radially offset with respect to the axis of said motor to an extent depending on the value of said rotational motor speed.
---
paper_title: A Novel Transductive SVM for Semisupervised Classification of Remote-Sensing Images

paper_content:

This paper introduces a semisupervised classification method that exploits both labeled and unlabeled samples for addressing ill-posed problems with support vector machines (SVMs). The method is based on recent developments in statistical learning theory concerning transductive inference and in particular transductive SVMs (TSVMs). TSVMs exploit specific iterative algorithms which gradually search a reliable separating hyperplane (in the kernel space) with a transductive process that incorporates both labeled and unlabeled samples in the training phase. Based on an analysis of the properties of the TSVMs presented in the literature, a novel modified TSVM classifier designed for addressing ill-posed remote-sensing problems is proposed. In particular, the proposed technique: 1) is based on a novel transductive procedure that exploits a weighting strategy for unlabeled patterns, based on a time-dependent criterion; 2) is able to mitigate the effects of suboptimal model selection (which is unavoidable in the presence of small-size training sets); and 3) can address multiclass cases. Experimental results confirm the effectiveness of the proposed method on a set of ill-posed remote-sensing classification problems representing different operative conditions
---
paper_title: Gaussian processes for machine learning

paper_content:

We give a basic introduction to Gaussian Process regression models. We focus on understanding the role of the stochastic process and how it is used to define a distribution over functions. We present the simple equations for incorporating training data and examine how to learn the hyperparameters using the marginal likelihood. We explain the practical advantages of Gaussian Process and end with conclusions and a look at the current trends in GP work.
---
paper_title: Estimating Biophysical Parameters from Remotely Sensed Imagery with Gaussian Processes

paper_content:

Recently, a new machine learning approach that is based on the Gaussian process (GP) theory has been introduced in the literature. According to this approach, the learning of a machine (regressor or classifier) is formulated in terms of a Bayesian estimation problem, where the parameters of the machine are assumed to be random variables which follow jointly a Gaussian distribution. The purpose of this work is to investigate this approach in the context of the estimation of biophysical parameters. Experimental results obtained on synthetic and real data, which simulate the spectral behavior of the chlorophyll concentration in subsurface waters, are reported and compared with those yielded by the general regression neural network (GRNN) and the epsiv-insensitive support vector regression (SVR) methods.
---
paper_title: Statistical learning theory

paper_content:

A comprehensive look at learning and generalization theory. The statistical theory of learning and generalization concerns the problem of choosing desired functions on the basis of empirical data. Highly applicable to a variety of computer science and robotics fields, this book offers lucid coverage of the theory as a whole. Presenting a method for determining the necessary and sufficient conditions for consistency of learning process, the author covers function estimates from small data pools, applying these estimations to real-life problems, and much more.
---
paper_title: Ocean color chlorophyll algorithms for SeaWiFS

paper_content:

A large data set containing coincident in situ chlorophyll and remote sensing reflectance measurements was used to evaluate the accuracy, precision, and suitability of a wide variety of ocean color chlorophyll algorithms for use by SeaWiFS (Sea-viewing Wide Field-of-view Sensor). The radiance-chlorophyll data were assembled from various sources during the SeaWiFS Bio-optical Algorithm Mini-Workshop (SeaBAM) and is composed of 919 stations encompassing chlorophyll concentrations between 0.019 and 32.79 μg L−1. Most of the observations are from Case I nonpolar waters, and ∼20 observations are from more turbid coastal waters. A variety of statistical and graphical criteria were used to evaluate the performances of 2 semianalytic and 15 empirical chlorophyll/pigment algorithms subjected to the SeaBAM data. The empirical algorithms generally performed better than the semianalytic. Cubic polynomial formulations were generally superior to other kinds of equations. Empirical algorithms with increasing complexity (number of coefficients and wavebands), were calibrated to the SeaBAM data, and evaluated to illustrate the relative merits of different formulations. The ocean chlorophyll 2 algorithm (OC2), a modified cubic polynomial (MCP) function which uses Rrs490/Rrs555, well simulates the sigmoidal pattern evident between log-transformed radiance ratios and chlorophyll, and has been chosen as the at-launch SeaWiFS operational chlorophyll a algorithm. Improved performance was obtained using the ocean chlorophyll 4 algorithm (OC4), a four-band (443, 490, 510, 555 nm), maximum band ratio formulation. This maximum band ratio (MBR) is a new approach in empirical ocean color algorithms and has the potential advantage of maintaining the highest possible satellite sensor signal: noise ratio over a 3-orders-of-magnitude range in chlorophyll concentration.
---
paper_title: Retrieval of sea water optically active parameters from hyperspectral data by means of generalized radial basis function neural networks

paper_content:

The authors present a new methodology for estimating the concentration of sea water optically active constituents from remotely sensed hyperspectral data, based on generalized radial basis function neural networks (GRBF-NNs). This family of NNs is particularly suited to approximate relationships like those between hyperspectral reflectance data and the concentrations of optically active constituents of the water body, which are highly nonlinear, especially in case II waters. Three main water constituents are taken into account: phytoplankton, nonchlorophyllous particles, and yellow substance. Each parameter is estimated by means of a specific multi-input single-output GRBF-NN. The authors adopt a recently proposed network learning strategy based on the combined use of the regression tree procedure and forward selection. The effectiveness of this approach, which is completely general and can be easily applied to any hyperspectral sensor, is proved using data simulated with an ocean color model over the channels of the medium resolution imaging spectrometer (MERIS), the new generation ESA sensor to be launched in 2001. The authors define the estimation algorithms over waters of cases I, II, and I+II and compare their performance with that of classical band-ratio, single-band, and multilinear algorithms. Generally, the GRBF-NN algorithms outperform the classical ones, except for the multilinear over case I waters. A particular improvement Is over case II waters, where the mean square error (MSE) can be reduced by one or two orders of magnitude over the error of multilinear and band-ratio algorithms, respectively.
---
paper_title: Principal Component Analysis

paper_content:

When large multivariate datasets are analyzed, it is often desirable to reduce their dimensionality. Principal component analysis is one technique for doing this. It replaces the p original variables by a smaller number, q, of derived variables, the principal components, which are linear combinations of the original variables. Often, it is possible to retain most of the variability in the original variables with q very much smaller than p. Despite its apparent simplicity, principal component analysis has a number of subtleties, and it has many uses and extensions. A number of choices associated with the technique are briefly discussed, namely, covariance or correlation, how many components, and different normalization constraints, as well as confusion with factor analysis. Various uses and extensions are outlined. ::: ::: ::: Keywords: ::: ::: dimension reduction; ::: factor analysis; ::: multivariate analysis; ::: variance maximization
---
paper_title: Kernel Methods for Pattern Analysis

paper_content:

Kernel methods provide a powerful and unified framework for pattern discovery, motivating algorithms that can act on general types of data (e.g. strings, vectors or text) and look for general types of relations (e.g. rankings, classifications, regressions, clusters). The application areas range from neural networks and pattern recognition to machine learning and data mining. This book, developed from lectures and tutorials, fulfils two major roles: firstly it provides practitioners with a large toolkit of algorithms, kernels and solutions ready to use for standard pattern discovery problems in fields such as bioinformatics, text analysis, image analysis. Secondly it provides an easy introduction for students and researchers to the growing field of kernel-based pattern analysis, demonstrating with examples how to handcraft an algorithm or a kernel for a new specific application, and covering all the necessary conceptual and mathematical tools to do so.
---
paper_title: Multivariate Data Analysis in Chemistry

paper_content:

Any data table produced in a chemical investigation can be analysed by bilinear projection methods, i. e. principal components and factor analysis and their extensions. Representing the table rows (objects) as points in a p-dimensional space, these methods project the point swarm of the data set or parts of it down on a F-dimensional subspace (plane or hyperplane). Different questions put to the data table correspond to different projections.
---
paper_title: Kernel Methods for Pattern Analysis

paper_content:

Kernel methods provide a powerful and unified framework for pattern discovery, motivating algorithms that can act on general types of data (e.g. strings, vectors or text) and look for general types of relations (e.g. rankings, classifications, regressions, clusters). The application areas range from neural networks and pattern recognition to machine learning and data mining. This book, developed from lectures and tutorials, fulfils two major roles: firstly it provides practitioners with a large toolkit of algorithms, kernels and solutions ready to use for standard pattern discovery problems in fields such as bioinformatics, text analysis, image analysis. Secondly it provides an easy introduction for students and researchers to the growing field of kernel-based pattern analysis, demonstrating with examples how to handcraft an algorithm or a kernel for a new specific application, and covering all the necessary conceptual and mathematical tools to do so.
---
paper_title: Gradient Optimization for multiple kernel's parameters in support vector machines classification

paper_content:

The subject of this work is the model selection of kernels with multiple parameters for support vector machines (SVM), with the purpose of classifying hyperspectral remote sensing data. During the training process, the kernel parameters need to be tuned properly. In this work a gradient descent based algorithm is used to estimate the parameters. The selection of multiple parameters is addressed, and an approach based on the analysis of the variance values of individual bands was proposed. Several state of the art kernels were tested. Experiments were conducted on real hyperspectral data. Results obtained with the different approaches/kernels were compared statistically, and showed good results in terms classification accuracies and processing time.
---
paper_title: Customizing Kernel Functions for SVM-Based Hyperspectral Image Classification

paper_content:

Previous research applying kernel methods such as support vector machines (SVMs) to hyperspectral image classification has achieved performance competitive with the best available algorithms. However, few efforts have been made to extend SVMs to cover the specific requirements of hyperspectral image classification, for example, by building tailor-made kernels. Observation of real-life spectral imagery from the AVIRIS hyperspectral sensor shows that the useful information for classification is not equally distributed across bands, which provides potential to enhance the SVM's performance through exploring different kernel functions. Spectrally weighted kernels are, therefore, proposed, and a set of particular weights is chosen by either optimizing an estimate of generalization error or evaluating each band's utility level. To assess the effectiveness of the proposed method, experiments are carried out on the publicly available 92AV3C dataset collected from the 220-dimensional AVIRIS hyperspectral sensor. Results indicate that the method is generally effective in improving performance: spectral weighting based on learning weights by gradient descent is found to be slightly better than an alternative method based on estimating "relevance" between band information and ground truth.
---
paper_title: Multisource Composite Kernels for Urban-Image Classification

paper_content:

This letter presents advanced classification methods for very high resolution images. Efficient multisource information, both spectral and spatial, is exploited through the use of composite kernels in support vector machines. Weighted summations of kernels accounting for separate sources of spectral and spatial information are analyzed and compared to classical approaches such as pure spectral classification or stacked approaches using all the features in a single vector. Model selection problems are addressed, as well as the importance of the different kernels in the weighted summation.
---
paper_title: Composite kernels for hyperspectral image classification

paper_content:

This letter presents a framework of composite kernel machines for enhanced classification of hyperspectral images. This novel method exploits the properties of Mercer's kernels to construct a family of composite kernels that easily combine spatial and spectral information. This framework of composite kernels demonstrates: 1) enhanced classification accuracy as compared to traditional approaches that take into account the spectral information only: 2) flexibility to balance between the spatial and spectral information in the classifier; and 3) computational efficiency. In addition, the proposed family of kernel classifiers opens a wide field for future developments in which spatial and spectral information can be easily integrated.
---
paper_title: Kernel-Based Framework for Multitemporal and Multisource Remote Sensing Data Classification and Change Detection

paper_content:

The multitemporal classification of remote sensing images is a challenging problem, in which the efficient combination of different sources of information (e.g., temporal, contextual, or multisensor) can improve the results. In this paper, we present a general framework based on kernel methods for the integration of heterogeneous sources of information. Using the theoretical principles in this framework, three main contributions are presented. First, a novel family of kernel-based methods for multitemporal classification of remote sensing images is presented. The second contribution is the development of nonlinear kernel classifiers for the well-known difference and ratioing change detection methods by formulating them in an adequate high-dimensional feature space. Finally, the presented methodology allows the integration of contextual information and multisensor images with different levels of nonlinear sophistication. The binary support vector (SV) classifier and the one-class SV domain description classifier are evaluated by using both linear and nonlinear kernel functions. Good performance on synthetic and real multitemporal classification scenarios illustrates the generalization of the framework and the capabilities of the proposed algorithms.
---
paper_title: A statistical framework for genomic data fusion

paper_content:

Motivation: During the past decade, the new focus on genomics has highlighted a particular challenge: to integrate the different views of the genome that are provided by various types of experimental data. ::: ::: Results: This paper describes a computational framework for integrating and drawing inferences from a collection of genome-wide measurements. Each dataset is represented via a kernel function, which defines generalized similarity relationships between pairs of entities, such as genes or proteins. The kernel representation is both flexible and efficient, and can be applied to many different types of data. Furthermore, kernel functions derived from different types of data can be combined in a straightforward fashion. Recent advances in the theory of kernel methods have provided efficient algorithms to perform such combinations in a way that minimizes a statistical loss function. These methods exploit semidefinite programming techniques to reduce the problem of finding optimizing kernel combinations to a convex optimization problem. Computational experiments performed using yeast genome-wide datasets, including amino acid sequences, hydropathy profiles, gene expression data and known protein--protein interactions, demonstrate the utility of this approach. A statistical learning algorithm trained from all of these data to recognize particular classes of proteins---membrane proteins and ribosomal proteins---performs significantly better than the same algorithm trained on any single type of data. ::: ::: Availability: Supplementary data at http://noble.gs.washington.edu/proj/sdp-svm
---
paper_title: Toward the Automatic Updating of Land-Cover Maps by a Domain-Adaptation SVM Classifier and a Circular Validation Strategy

paper_content:

In this paper, we address automatic updating of land-cover maps by using remote-sensing images periodically acquired over the same investigated area under the hypothesis that a reliable ground truth is not available for all the considered acquisitions. The problem is modeled in the domain-adaptation framework by introducing a novel method designed for land-cover map updating, which is based on a domain-adaptation support vector machine technique. In addition, a novel circular accuracy assessment strategy is proposed for the validation of the results obtained by domain-adaptation classifiers when no ground-truth labels for the considered image are available. Experimental results obtained on a multitemporal and multispectral data set confirmed the effectiveness and the reliability of the proposed system.
---
paper_title: A Multiple Cascade-Classifier System for a Robust and Partially Unsupervised Updating of Land-Cover Maps

paper_content:

A system for a regular updating of land-cover maps is proposed that is based on the use of multitemporal remote sensing images. Such a system is able to address the updating problem under the realistic but critical constraint that, for the image to be classified (i.e., the most recent of the considered multitemporal dataset) no ground truth information is available. The system is composed of an ensemble of partially unsupervised classifiers integrated in a multiple-classifier architecture. Each classifier of the ensemble exhibits the following novel characteristics: (1) it is developed in the framework of the cascade-classification approach to exploit the temporal correlation existing between images acquired at different times in the considered area; and (2) it is based on a partially unsupervised methodology capable of accomplishing the classification process under the aforementioned critical constraint. Both a parametric maximum-likelihood (ML) classification approach and a nonparametric radial basis function (RBF) neural-network classification approach are used as basic methods for the development of partially unsupervised cascade classifiers. In addition, in order to generate an effective ensemble of classification algorithms, hybrid ML and RBF neural-network cascade classifiers are defined by exploiting the characteristics of the cascade-classification methodology. The results yielded by the different classifiers are combined by using standard unsupervised combination strategies. This allows the definition of a robust and accurate partially unsupervised classification system capable of analyzing a wide typology of remote sensing data (e.g., images acquired by passive sensors, synthetic aperture radar images, and multisensor and multisource data). Experimental results obtained on a real multitemporal and multisource dataset confirm the effectiveness of the proposed system.
---
paper_title: Mean Map Kernel Methods for Semisupervised Cloud Classification

paper_content:

Remote sensing image classification constitutes a challenging problem since very few labeled pixels are typically available from the analyzed scene. In such situations, labeled data extracted from other images modeling similar problems might be used to improve the classification accuracy. However, when training and test samples follow even slightly different distributions, classification is very difficult. This problem is known as sample selection bias. In this paper, we propose a new method to combine labeled and unlabeled pixels to increase classification reliability and accuracy. A semisupervised support vector machine classifier based on the combination of clustering and the mean map kernel is proposed. The method reinforces samples in the same cluster belonging to the same class by combining sample and cluster similarities implicitly in the kernel space. A soft version of the method is also proposed where only the most reliable training samples, in terms of likelihood of the image data distribution, are used. Capabilities of the proposed method are illustrated in a cloud screening application using data from the MEdium Resolution Imaging Spectrometer (MERIS) instrument onboard the European Space Agency ENVISAT satellite. Cloud screening constitutes a clear example of sample selection bias since cloud features change to a great extent depending on the cloud type, thickness, transparency, height, and background. Good results are obtained and show that the method is particularly well suited for situations where the available labeled information does not adequately describe the classes in the test data.
---
paper_title: Unsupervised retraining of a maximum likelihood classifier for the analysis of multitemporal remote sensing images

paper_content:

An unsupervised retraining technique for a maximum likelihood (ML) classifier is presented. The proposed technique allows the classifier's parameters, obtained by supervised learning on a specific image, to be updated in a totally unsupervised way on the basis of the distribution of a new image to be classified. This enables the classifier to provide a high accuracy for the new image even when the corresponding training set is not available.
---
paper_title: Learning to localize objects with structured output regression

paper_content:

Sliding window classifiers are among the most successful and widely applied techniques for object localization. However, training is typically done in a way that is not specific to the localization task. First a binary classifier is trained using a sample of positive and negative examples, and this classifier is subsequently applied to multiple regions within test images. We propose instead to treat object localization in a principled way by posing it as a problem of predicting structured data: we model the problem not as binary classification, but as the prediction of the bounding box of objects located in images. The use of a joint-kernelframework allows us to formulate the training procedure as a generalization of an SVM, which can be solved efficiently. We further improve computational efficiency by using a branch-and-bound strategy for localization during both training and testing. Experimental evaluation on the PASCAL VOC and TU Darmstadt datasets show that the structured training procedure improves performance over binary training as well as the best previously published scores.
---
paper_title: An Active Learning Approach to Hyperspectral Data Classification

paper_content:

Obtaining training data for land cover classification using remotely sensed data is time consuming and expensive especially for relatively inaccessible locations. Therefore, designing classifiers that use as few labeled data points as possible is highly desirable. Existing approaches typically make use of small-sample techniques and semisupervision to deal with the lack of labeled data. In this paper, we propose an active learning technique that efficiently updates existing classifiers by using fewer labeled data points than semisupervised methods. Further, unlike semisupervised methods, our proposed technique is well suited for learning or adapting classifiers when there is substantial change in the spectral signatures between labeled and unlabeled data. Thus, our active learning approach is also useful for classifying a series of spatially/temporally related images, wherein the spectral signatures vary across the images. Our interleaved semisupervised active learning method was tested on both single and spatially/temporally related hyperspectral data sets. We present empirical results that establish the superior performance of our proposed approach versus other active learning and semisupervised methods.
---
paper_title: Segmentation of multispectral remote sensing images using active support vector machines

paper_content:

The problem of scarcity of labeled pixels, required for segmentation of remotely sensed satellite images in supervised pixel classification framework, is addressed in this article. A support vector machine (SVM) is considered for classifying the pixels into different landcover types. It is initially designed using a small set of labeled points, and subsequently refined by actively querying for the labels of pixels from a pool of unlabeled data. The label of the most interesting/ ambiguous unlabeled point is queried at each step. Here, active learning is exploited to minimize the number of labeled data used by the SVM classifier by several orders. These features are demonstrated on an IRS-1A four band multi-spectral image. Comparison with related methods is made in terms of number of data points used, computational time and a cluster quality measure.
---
paper_title: Geostatistical and texture analysis of airborne-acquired images used in forest classification

paper_content:

Airborne sensor image texture derived following a geostatistical analysis can increase the accuracy of forest classification because the resulting texture is insensitive to random variations in spectral response but related to the structural features of interest at the scale of a forest inventory (e.g. tree species). The combination of spectral and textural data derived from a kriging surface provided 86% classification accuracy in 36 pure and mixed-wood stands in seven forest classes in Alberta. This is an increase over the classification accuracy obtained when texture was derived from the original image data, and when the spectral response patterns were used alone.
---
paper_title: Detection of Unexploded Ordnance via Efficient Semisupervised and Active Learning

paper_content:

Semi supervised learning and active learning are considered for unexploded ordnance (UXO) detection. Semi supervised learning algorithms are designed using both labeled and unlabeled data, where here labeled data correspond to sensor signatures for which the identity of the buried item (UXO/non-UXO) is known; for unlabeled data, one only has access to the corresponding sensor data. Active learning is used to define which unlabeled signatures would be most informative to improve the classifier design if the associated label could be acquired (where for UXO sensing, the label is acquired by excavation). A graph-based semi supervised algorithm is applied, which employs the idea of a random Markov walk on a graph, thereby exploiting knowledge of the data manifold (where the manifold is defined by both the labeled and unlabeled data). The algorithm is used to infer labels for the unlabeled data, providing a probability that a given unlabeled signature corresponds to a buried UXO. An efficient active-learning procedure is developed for this algorithm, based on a mutual information measure. In this manner, one initially performs excavation with the purpose of acquiring labels to improve the classifier, and once this active-learning phase is completed, the resulting semi supervised classifier is then applied to the remaining unlabeled signatures to quantify the probability that each such item is a UXO. Example classification results are presented for an actual UXO site, based on electromagnetic induction and magnetometer data. Performance is assessed in comparison to other semi supervised approaches, as well as to supervised algorithms.
---
paper_title: Active Learning Methods for Remote Sensing Image Classification

paper_content:

In this paper, we propose two active learning algorithms for semiautomatic definition of training samples in remote sensing image classification. Based on predefined heuristics, the classifier ranks the unlabeled pixels and automatically chooses those that are considered the most valuable for its improvement. Once the pixels have been selected, the analyst labels them manually and the process is iterated. Starting with a small and nonoptimal training set, the model itself builds the optimal set of samples which minimizes the classification error. We have applied the proposed algorithms to a variety of remote sensing data, including very high resolution and hyperspectral images, using support vector machines. Experimental results confirm the consistency of the methods. The required number of training samples can be reduced to 10% using the methods proposed, reaching the same level of accuracy as larger data sets. A comparison with a state-of-the-art active learning method, margin sampling, is provided, highlighting advantages of the methods proposed. The effect of spatial resolution and separability of the classes on the quality of the selection of pixels is also discussed.
---
paper_title: A Parallel Processing Algorithm for Remote Sensing Classification

paper_content:

A current thread in parallel computation is the use of cluster computers created by networking a few to thousands of commodity general-purpose workstation-level commuters using the Linux operating system. For example on the Medusa cluster at NASA/GSFC, this provides for super computing performance, 130 Gflops (Linpack Benchmark) at moderate cost, $370K. However, to be useful for scientific computing in the area of Earth science, issues of ease of programming, access to existing scientific libraries, and portability of existing code need to be considered. In this paper, I address these issues in the context of tools for rendering earth science remote sensing data into useful products. In particular, I focus on a problem that can be decomposed into a set of independent tasks, which on a serial computer would be performed sequentially, but with a cluster computer can be performed in parallel, giving an obvious speedup. To make the ideas concrete, I consider the problem of classifying hyperspectral imagery where some ground truth is available to train the classifier. In particular I will use the Support Vector Machine (SVM) approach as applied to hyperspectral imagery [16].
---
paper_title: Atmospheric And Environmental Remote Sensing Data Processing And Utilization: An End-to-end System Perspective

paper_content:

Proceedings of SPIE present the original research papers presented at SPIE conferences and other high-quality conferences in the broad-ranging fields of optics and photonics. These books provide prompt access to the latest innovations in research and technology in their respective fields. Proceedings of SPIE are among the most cited references in patent literature.
---
paper_title: High Performance Computing in Remote Sensing

paper_content:

Solutions for Time-Critical Remote Sensing Applications The recent use of latest-generation sensors in airborne and satellite platforms is producing a nearly continual stream of high-dimensional data, which, in turn, is creating new processing challenges. To address the computational requirements of time-critical applications, researchers have begun incorporating high performance computing (HPC) models in remote sensing missions. High Performance Computing in Remote Sensing is one of the first volumes to explore state-of-the-art HPC techniques in the context of remote sensing problems. It focuses on the computational complexity of algorithms that are designed for parallel computing and processing. A Diverse Collection of Parallel Computing Techniques and Architectures The book first addresses key computing concepts and developments in remote sensing. It also covers application areas not necessarily related to remote sensing, such as multimedia andvideo processing. Each subsequent chapter illustrate s a specific parallel computing paradigm, including multiprocessor (cluster-based) systems, large-scale and heterogeneous networks of computers, grid computing platforms, and specialized hardware architectures for remotely sensed data analysis and interpretation. An Interdisciplinary Forum to Encourage Novel Ideas The extensive reviews of current and future developments combined with thoughtful perspectives on the potential challenges of adapting HPC paradigms to remote sensing problems will undoubtedly foster collaboration and development among many fields.
---

Title: A Review of Kernel Methods in Remote Sensing Data Analysis
Section 1: Introduction
Description 1: Provide an overview of the significance of remotely sensed images in Earth Observation and the complexity introduced by high-resolution data, alongside the motivation for using kernel methods in this field.

Section 2: Classification with Kernels
Description 2: Discuss the application of kernel methods in image classification, specifically the use of support vector machines (SVMs) and their advantages in handling high-dimensional data, incomplete knowledge, and semisupervised learning.

Section 3: Model Inversion with Kernels
Description 3: Explain the role of kernel methods in inverting forward models to predict physical parameters from spectral data, a crucial task in remote sensing.

Section 4: Introduction to Kernel Methods
Description 4: Introduce basic kernel methods concepts, including notation, properties, and the mathematical foundations derived from linear algebra and functional analysis.

Section 5: Measuring Similarity with Kernels
Description 5: Describe the significance of defining similarity between samples in kernel methods, including mapping into high-dimensional spaces and use of kernel functions.

Section 6: Positive Definite Kernels
Description 6: Define positive definite kernels and discuss their importance in ensuring the validity of kernel-based algorithms.

Section 7: Basic Operations with Kernels
Description 7: Review fundamental operations that can be performed with kernels, such as translation, centering, subspace projections, computing distances, and normalization.

Section 8: Standard Kernels
Description 8: Discuss common types of kernels used in practice, like linear, polynomial, and radial basis function (RBF) kernels, and their practical advantages.

Section 9: Kernel Development
Description 9: Address how new kernels can be developed by combining basic kernels and manipulating kernel functions to better fit specific remote sensing data characteristics.

Section 10: Kernel methods in remote sensing data classification
Description 10: Highlight practical applications and success stories of kernel methods, especially SVMs, in various remote sensing data classification tasks.

Section 11: Support Vector Machine (SVM)
Description 11: Detail the formulation and application of SVM for remote sensing data, explaining concepts such as dual problem, decision functions, and variations like ν-SVM.

Section 12: Support Vector Data Description (SVDD)
Description 12: Introduce SVDD for one-class classification problems and outline its methodology and applications.

Section 13: One-class Support Vector Machine (OC-SVM)
Description 13: Compare OC-SVM with SVDD, explain its formulation, and discuss situations where it can be particularly useful.

Section 14: Kernel Fisher's Discriminant (KFD)
Description 14: Explain the concept of Kernel Fisher's Discriminant, detailing its application for maximizing class separability in a high-dimensional space.

Section 15: Experimental Results for Supervised Classification
Description 15: Present experimental results comparing different supervised kernel-based classifiers in a remote sensing context, focusing on classification accuracy and robustness.

Section 16: Semisupervised Image Classification
Description 16: Discuss the application of semisupervised learning approaches in remote sensing and their potential to improve classification accuracy using both labeled and unlabeled data.

Section 17: Manifold-based Regularization Framework
Description 17: Explain the importance of regularization in preventing overfitting and describe how manifold-based regularization can be used in image classification.

Section 18: Semisupervised Regularization Framework
Description 18: Introduce the framework for regularization with unlabeled samples and its importance in remote sensing data analysis.

Section 19: Laplacian Support Vector Machine (LapSVM)
Description 19: Review the Laplacian SVM approach, its implementation, and its application in remote sensing data classification.

Section 20: Transductive SVM (TSVM)
Description 20: Describe the concept and methodology of TSVM, focusing on its application and advantages in remote sensing.

Section 21: Experimental Results for Semisupervised Classification
Description 21: Provide experimental results for semisupervised methods, comparing their performance in remote sensing applications.

Section 22: Kernel methods in biophysical parameter estimation
Description 22: Discuss the need for robust regression tools in estimating biophysical parameters and the role of kernel methods in this task.

Section 23: Support Vector Regression (SVR)
Description 23: Detail the SVR methodology for regression problems in remote sensing, explaining the ε-insensitive cost function and its implementation.

Section 24: Relevance Vector Machines (RVM)
Description 24: Introduce RVMs, compare them with SVRs, and explain their advantages in handling some of the limitations of SVRs.

Section 25: Gaussian Processes (GP)
Description 25: Explain the use of Gaussian Processes for regression in remote sensing, highlighting their Bayesian approach and advantages.

Section 26: Experimental Results
Description 26: Compare the performance of SVR, RVM, and GP in biophysical parameter estimation with experimental results.

Section 27: Kernel Methods for Feature extraction
Description 27: Discuss the importance of feature extraction in high-dimensional data analysis and the role of kernel methods in linear and nonlinear feature extraction.

Section 28: Mutivariate Analysis Methods (MVA)
Description 28: Introduce multivariate analysis methods for feature extraction, focusing on their applications in remote sensing.

Section 29: Principal Component Analysis (PCA)
Description 29: Explain the concept of PCA, its implementation, and its limitations in remote sensing data analysis.

Section 30: Partial Least Squares (PLS)
Description 30: Introduce PLS and discuss how it can be used to find directions of maximum covariance between inputs and targets.

Section 31: Kernel Multivariate Analysis (KMVA)
Description 31: Describe how kernel methods can be used to implement nonlinear versions of traditional multivariate analysis methods.

Section 32: Kernel Principal Component Analysis (KPCA)
Description 32: Explain the methodology of KPCA and its application for finding directions of maximum variance in high-dimensional spaces.

Section 33: Kernel Partial Least Squares (KPLS)
Description 33: Detail the KPLS method for finding directions of maximum covariance in the feature space and its application.

Section 34: Future Trends in Remote Sensing Kernel Learning
Description 34: Outline emerging trends and potential future research directions in kernel-based learning methodologies for remote sensing.

Section 35: Multiple Kernel Learning
Description 35: Discuss the concept and applications of multiple kernel learning in remote sensing to combine information from different sources and sensors.

Section 36: Domain Learning
Description 36: Describe the challenges of domain adaptation and transfer learning in remote sensing and how kernel methods can address these issues.

Section 37: Structured Learning
Description 37: Introduce structured learning and its potential applications in remote sensing for handling complex output spaces.

Section 38: Active Learning
Description 38: Explain active learning methodologies and their significance in optimizing training sets for remote sensing applications.

Section 39: Parallel Implementations
Description 39: Discuss the scalability issue with kernel methods in remote sensing and explore parallel processing solutions to address this problem.

Section 40: Conclusions
Description 40: Summarize the key points discussed in the review, emphasizing the benefits and future potential of kernel methods in remote sensing data analysis.