A review of Convolutional Neural Networks in Remote Sensing Image
---
paper_title: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

paper_content:

State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet [1] and Fast R-CNN [2] have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features—using the recently popular terminology of neural networks with ’attention’ mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model [3] , our detection system has a frame rate of 5 fps ( including all steps ) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available.
---
paper_title: Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations

paper_content:

There has been much interest in unsupervised learning of hierarchical generative models such as deep belief networks. Scaling such models to full-sized, high-dimensional images remains a difficult problem. To address this problem, we present the convolutional deep belief network, a hierarchical generative model which scales to realistic image sizes. This model is translation-invariant and supports efficient bottom-up and top-down probabilistic inference. Key to our approach is probabilistic max-pooling, a novel technique which shrinks the representations of higher layers in a probabilistically sound way. Our experiments show that the algorithm learns useful high-level visual features, such as object parts, from unlabeled images of objects and natural scenes. We demonstrate excellent performance on several visual recognition tasks and show that our model can perform hierarchical (bottom-up and top-down) inference over full-sized images.
---
paper_title: Backpropagation Applied to Handwritten Zip Code Recognition

paper_content:

The ability of learning networks to generalize can be greatly enhanced by providing constraints from the task domain. This paper demonstrates how such constraints can be integrated into a backpropagation network through the architecture of the network. This approach has been successfully applied to the recognition of handwritten zip code digits provided by the U.S. Postal Service. A single network learns the entire recognition operation, going from the normalized image of the character to the final classification.
---
paper_title: A Fast Learning Algorithm for Deep Belief Nets

paper_content:

We show how to use "complementary priors" to eliminate the explaining-away effects that make inference difficult in densely connected belief nets that have many hidden layers. Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory. The fast, greedy algorithm is used to initialize a slower learning procedure that fine-tunes the weights using a contrastive version of the wake-sleep algorithm. After fine-tuning, a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels. This generative model gives better digit classification than the best discriminative learning algorithms. The low-dimensional manifolds on which the digits lie are modeled by long ravines in the free-energy landscape of the top-level associative memory, and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind.
---
paper_title: ImageNet classification with deep convolutional neural networks

paper_content:

We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0%, respectively, which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully connected layers we employed a recently developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.
---
paper_title: Very Deep Convolutional Networks for Large-Scale Image Recognition

paper_content:

In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.
---
paper_title: Improving neural networks by preventing co-adaptation of feature detectors

paper_content:

When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data. This "overfitting" is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate. Random "dropout" gives big improvements on many benchmark tasks and sets new records for speech and object recognition.
---
paper_title: Regularization of neural networks using dropconnect

paper_content:

We introduce DropConnect, a generalization of Dropout (Hinton et al., 2012), for regularizing large fully-connected layers within neural networks. When training with Dropout, a randomly selected subset of activations are set to zero within each layer. DropConnect instead sets a randomly selected subset of weights within the network to zero. Each unit thus receives input from a random subset of units in the previous layer. We derive a bound on the generalization performance of both Dropout and DropConnect. We then evaluate DropConnect on a range of datasets, comparing to Dropout, and show state-of-the-art results on several image recognition benchmarks by aggregating multiple DropConnect-trained models.
---
paper_title: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

paper_content:

Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters.
---
paper_title: On automated source selection for transfer learning in convolutional neural networks

paper_content:

Abstract Transfer learning, or inductive transfer, refers to the transfer of knowledge from a source task to a target task. In the context of convolutional neural networks (CNNs), transfer learning can be implemented by transplanting the learned feature layers from one CNN (derived from the source task) to initialize another (for the target task). Previous research has shown that the choice of the source CNN impacts the performance of the target task. In the current literature, there is no principled way for selecting a source CNN for a given target task despite the increasing availability of pre-trained source CNNs. In this paper we investigate the possibility of automatically ranking source CNNs prior to utilizing them for a target task. In particular, we present an information theoretic framework to understand the source-target relationship and use this as a basis to derive an approach to automatically rank source CNNs in an efficient, zero-shot manner. The practical utility of the approach is thoroughly evaluated using the Places-MIT dataset, MNIST dataset and a real-world MRI database. Experimental results demonstrate the efficacy of the proposed ranking method for transfer learning.
---
paper_title: Hyperspectral remote sensing image change detection based on tensor and deep learning

paper_content:

Abstract Considering the bottleneck in improving the performance of the existing multi-temporal hyperspectral remote sensing (HSRS) image change detection methods, a HSRS image change detection solution based on tensor and deep learning is proposed in this study. At first, a tensor-based information model (TFS-Cube) of underlying features change in HSRS images is established. The wavelet texture feature change, spectral feature change and spatio-temporal autocorrelation coefficient of different-temporal related pixels are combined with three-order tensor, so as to make full use of the underlying features change information of HSRS images, optimize the organization mode and maintain the integrity of constraints between different underlying features. Secondly, a restricted Boltzmann Machine based on three-order tensor (Tensor3-RBM) is designed. The input, output and unsupervised learning of TFS-Cube tensor data are realized by multi-linear operations in Tensor3-RBMs. A large number of unlabeled samples are trained layer by layer through multilayer Tensor3-RBMs. Finally, the traditional BP neural network on the top layer of deep belief network (DBN) is replaced with support tensor machine (STM), and a deep belief network with multi-layer Tensor3-RBM and STM (TRS-DBN) is constructed. A small number of labeled samples are used for supervised learning and TRS-DBN global parameters optimization to improve the accuracy of TRS-DBN change detection. Two types of HSRS images from different sensors, AVIRIS and EO-1 Hyperion, are used as the data sources (double-temporal). Four representative experimental regions are randomly selected from the two areas covered by AVIRIS and EO-1 Hyperion HSRS images respectively (two regions in each area) to detect the land use changes. Experimental results demonstrate that TRS-DBN has higher change detection accuracy than similar methods and a good automation level.
---
paper_title: Deep Convolutional Neural Networks for Hyperspectral Image Classification

paper_content:

Recently, convolutional neural networks have demonstrated excellent performance on various visual tasks, including the classification of common two-dimensional images. In this paper, deep convolutional neural networks are employed to classify hyperspectral images directly in spectral domain. More specifically, the architecture of the proposed classifier contains five layers with weights which are the input layer, the convolutional layer, the max pooling layer, the full connection layer, and the output layer. These five layers are implemented on each spectral signature to discriminate against others. Experimental results based on several hyperspectral image data sets demonstrate that the proposed method can achieve better classification performance than some traditional methods, such as support vector machines and the conventional deep learning-based methods.
---
paper_title: Building Extraction in Very High Resolution Remote Sensing Imagery Using Deep Learning and Guided Filters

paper_content:

Very high resolution (VHR) remote sensing imagery has been used for land cover classification, and it tends to a transition from land-use classification to pixel-level semantic segmentation. Inspired by the recent success of deep learning and the filter method in computer vision, this work provides a segmentation model, which designs an image segmentation neural network based on the deep residual networks and uses a guided filter to extract buildings in remote sensing imagery. Our method includes the following steps: first, the VHR remote sensing imagery is preprocessed and some hand-crafted features are calculated. Second, a designed deep network architecture is trained with the urban district remote sensing image to extract buildings at the pixel level. Third, a guided filter is employed to optimize the classification map produced by deep learning; at the same time, some salt-and-pepper noise is removed. Experimental results based on the Vaihingen and Potsdam datasets demonstrate that our method, which benefits from neural networks and guided filtering, achieves a higher overall accuracy when compared with other machine learning and deep learning methods. The method proposed shows outstanding performance in terms of the building extraction from diversified objects in the urban district.
---
paper_title: Convolutional Neural Networks for Large-Scale Remote-Sensing Image Classification

paper_content:

We propose an end-to-end framework for the dense, pixelwise classification of satellite imagery with convolutional neural networks (CNNs). In our framework, CNNs are directly trained to produce classification maps out of the input images. We first devise a fully convolutional architecture and demonstrate its relevance to the dense classification problem. We then address the issue of imperfect training data through a two-step training approach: CNNs are first initialized by using a large amount of possibly inaccurate reference data, and then refined on a small amount of accurately labeled data. To complete our framework, we design a multiscale neuron module that alleviates the common tradeoff between recognition and precise localization. A series of experiments show that our networks consider a large amount of context to provide fine-grained classification maps.
---
paper_title: Deep Learning for Satellite Image Classification

paper_content:

Nowadays, large amounts of high resolution remote-sensing images are acquired daily. However, the satellite image classification is requested for many applications such as modern city planning, agriculture and environmental monitoring. Many researchers introduce and discuss this domain but still, the sufficient and optimum degree has not been reached yet. Hence, this article focuses on evaluating the available and public remote-sensing datasets and common different techniques used for satellite image classification. The existing remote-sensing classification methods are categorized into four main categories according to the features they use: manually feature-based methods, unsupervised feature learning methods, supervised feature learning methods, and object-based methods. In recent years, there has been an extensive popularity of supervised deep learning methods in various remote-sensing applications, such as geospatial object detection and land use scene classification. Thus, the experiments, in this article, carried out on one of the popular deep learning models, Convolution Neural Networks (CNNs), precisely AlexNet architecture on a standard sounded dataset, UC-Merceed Land Use. Finally, a comparison with other different techniques is introduced.
---
paper_title: Deep learning with multi-scale feature fusion in remote sensing for automatic oceanic eddy detection

paper_content:

Abstract Oceanic eddies are ubiquitous in global oceans and play a major role in ocean energy transfer and nutrients distribution, thus being significant for understanding ocean current circulation and marine climate change. They are characterized by a combination of high-speed vertical rotations and horizontal movements, leading to irregular three-dimensional spiral structures. While the ability to detect eddies automatically and remotely is crucial to monitoring important spatial–temporal dynamics, existing methods are inaccurate because eddies are highly dynamic and the underlying physical processes are not well understood. Typically, remote sensing is used to detect eddies based on physical parameters, geometrics or other handcrafted features. In this paper, we show how Deep Learning may be used to reliably extract higher-level features and then fuse multi-scale features to identify eddies, regardless of their structures and scales. We learn eddy features using two principal component analysis convolutional layers, then perform a non-linear transformation of the features through a binary hashing layer and block-wise histograms. To handle the difficult problem of spatial variability across synthetic aperture radar (SAR) images, we introduce a spatial pyramid model to allow multi-scale features fusion. Finally, a linear support vector machine classifier recognizes the eddies. Our method, dubbed DeepEddy, is benchmarked against a dataset of 20,000 SAR image samples, achieving a 97.8 ± 1% accuracy of detection.
---
paper_title: Deep Learning Classification of Land Cover and Crop Types Using Remote Sensing Data

paper_content:

Deep learning (DL) is a powerful state-of-the-art technique for image processing including remote sensing (RS) images. This letter describes a multilevel DL architecture that targets land cover and crop type classification from multitemporal multisource satellite imagery. The pillars of the architecture are unsupervised neural network (NN) that is used for optical imagery segmentation and missing data restoration due to clouds and shadows, and an ensemble of supervised NNs. As basic supervised NN architecture, we use a traditional fully connected multilayer perceptron (MLP) and the most commonly used approach in RS community random forest, and compare them with convolutional NNs (CNNs). Experiments are carried out for the joint experiment of crop assessment and monitoring test site in Ukraine for classification of crops in a heterogeneous environment using nineteen multitemporal scenes acquired by Landsat-8 and Sentinel-1A RS satellites. The architecture with an ensemble of CNNs outperforms the one with MLPs allowing us to better discriminate certain summer crop types, in particular maize and soybeans, and yielding the target accuracies more than 85% for all major crops (wheat, maize, sunflower, soybeans, and sugar beet).
---

Title: A Review of Convolutional Neural Networks in Remote Sensing Image

Section 1: INTRODUCTION
Description 1: This section provides an overview of the significance of remote sensing images and the challenges associated with their analysis. It also introduces the development of CNNs and their relevance to remote sensing.

Section 2: CNN
Description 2: This section introduces Convolutional Neural Networks (CNNs) as the most representative supervised deep learning model and discusses the basic structure of CNNs.

Section 3: Structure of Basic CNN
Description 3: This section describes the basic architecture of CNNs, including input layers, convolutional layers, pooling layers, fully connected layers, and output layers, along with the mechanisms used for training.

Section 4: Classical CNN architectures
Description 4: This section provides summaries of three popular CNN architectures: AlexNet, GoogLeNet, and VGGNet, highlighting their structures, innovations, and performance in competitions.

Section 5: PROBLEMS IN CNN
Description 5: This section discusses the two main issues associated with CNNs: over-fitting and high commutation problems during the training process. It explores methods like data augmentation, Dropout, DropConnect, Transfer Learning, and Batch Normalization to address these issues.

Section 6: APPLICATION IN REMOTE SENSORNING
Description 6: This section reviews the application of CNNs in remote sensing by analyzing ten representative articles. It compares their approaches, datasets, network models, and evaluations, emphasizing image classification, object detection, and image fusion.

Section 7: CONCLUSION AND FUTURE WORK
Description 7: This section concludes the review by summarizing the successful applications of CNNs in remote sensing and suggests future research directions, such as efficient learning with small samples and optimization of network structures.