Video Processing From Electro-Optical Sensors for Object Detection and Tracking in a Maritime Environment: A Survey
---
paper_title: Looking at Vehicles on the Road: A Survey of Vision-Based Vehicle Detection, Tracking, and Behavior Analysis

paper_content:

This paper provides a review of the literature in on-road vision-based vehicle detection, tracking, and behavior understanding. Over the past decade, vision-based surround perception has progressed from its infancy into maturity. We provide a survey of recent works in the literature, placing vision-based vehicle detection in the context of sensor-based on-road surround analysis. We detail advances in vehicle detection, discussing monocular, stereo vision, and active sensor-vision fusion for on-road vehicle detection. We discuss vision-based vehicle tracking in the monocular and stereo-vision domains, analyzing filtering, estimation, and dynamical models. We discuss the nascent branch of intelligent vehicles research concerned with utilizing spatiotemporal measurements, trajectories, and various features to characterize on-road behavior. We provide a discussion on the state of the art, detail common performance metrics and benchmarks, and provide perspective on future research directions in the field.
---
paper_title: Stereovision based obstacle detection system for unmanned surface vehicle

paper_content:

This paper presents the stereo based obstacle detection system for unmanned surface vehicle (USV). The system is designed and developed toward the aim of real-time and robust obstacle detection and tracking at sea. Stereo vision methods and techniques have been employed to offer the capacity of detecting, locating and tracking multiple obstacles in the near field. Field test in the real scenes has been conducted, and the obstacle detection system for USV is proven to provide stable and satisfactory performance. The valid range with good accuracy of depth estimation is from 20 to 200 meters for high speed USV.
---
paper_title: Passive target tracking of marine traffic ships using onboard monocular camera for unmanned surface vessel

paper_content:

Enhancing the performance of passive target tracking and trajectory estimation of marine traffic ships is focused using a monocular camera mounted on an unmanned surface vessel. To accurately estimate the trajectory of a target traffic ship, the relative bearing and range information between the observing ship and the target ship is required. Monocular vision provides bearing information with reasonable accuracy but with no explicit range information. The relative range information can be extracted from the bearing changes induced by the relative ship motion in the framework of bearings-only tracking (BOT). BOT can be effective in crossing situations with large bearing angle changes. However, it often fails in head-on or overtaking situations due to small bearing angle changes and the resulting low observability of the tracking filter. To deal with the lack of observability, the vertical pixel distance between the horizon and the target ship in the image is used, which improves the overall target tracking performance. The feasibility and performance of the proposed tracking approach were validated through field experiments at sea.
---
paper_title: Argos - a Video Surveillance System for boat Traffic Monitoring in Venice

paper_content:

Visual surveillance in dynamic scenes is currently one of the most active research topics in computer vision, many existing applications are available. However, difficulties in realizing effective video surveillance systems that are robust to the many different conditions that arise in real environments, make the actual deployment of such systems very challenging. In this article, we present a real, unique and pioneer video surveillance system for boat traffic monitoring, ARGOS. The system runs continuously 24 hours a day, 7 days a week, day and night in the city of Venice (Italy) since 2007 and it is able to build a reliable background model of the water channel and to track the boats navigating the channel with good accuracy in real-time. A significant experimental evaluation, reported in this article, has been performed in order to assess the real performance of the system.
---
paper_title: Multispectral Target Detection and Tracking for Seaport Video Surveillance

paper_content:

In this paper, a video surveillance process is presented including target detection and tracking of ships at the entrance of a seaport in order to improve security and to prevent terrorist attacks. This process is helpful in the automatic analysis of movements inside the seaport. Steps of detection and tracking are completed using IR data whereas the pattern recognition stage is achieved on color data. A comparison of results of detection and tracking is presented on both IR and color data in order to justify the choice of IR images for these two steps. A draft description of the pattern recognition stage is finally drawn up as development prospect.
---
paper_title: Attenuation of Electromagnetic Radiation by Haze, Fog, Clouds, and Rain

paper_content:

Abstract : The report assembles, under one cover, the values of aerosol attenuation coefficients of regions in the electromagnetic (EM) spectrum containing so-called 'atmospheric windows,' in which EM radiation suffers the least amount of atmospheric gaseous absorption. The purpose is to enable rapid quantitative assessment of target acquisition terminal guidance sensors using the windows during adverse weather. Both calculated and available measured values are presented. Being a compilation drawn from numerous sources, the report is intended more as a handbook for ready use than as a theoretical treatise.
---
paper_title: SITUATION AWARENESS IN REMOTE CONTROL CENTRES FOR UNMANNED SHIPS

paper_content:

The feasibility of unmanned, autonomous merchant vessels is investigated by the EU project MUNIN (Maritime Unmanned Navigation through Intelligence in Networks). The ships will be manned during passage to and from port and unmanned during ocean-passage. When unmanned, the ships will be controlled by an automatic system informed by onboard sensors allowing the ship to make standard collision avoidance manoeuvres according to international regulation. The ship will be continuously monitored by a remote shore centre able to take remote control should the automatic systems falter. For the humans in the shore control centre the usual problems of automations remains as well as a pronounced problem of keeping up adequate situation awareness through remote sensing.
---
paper_title: Challenges in video based object detection in maritime scenario using computer vision

paper_content:

This paper discusses the technical challenges in maritime image processing and machine vision problems for video streams generated by cameras. Even well documented problems of horizon detection and registration of frames in a video are very challenging in maritime scenarios. More advanced problems of background subtraction and object detection in video streams are very challenging. Challenges arising from the dynamic nature of the background, unavailability of static cues, presence of small objects at distant backgrounds, illumination effects, all contribute to the challenges as discussed here.
---
paper_title: Integrated visual information for maritime surveillance

paper_content:

The main contribution of this chapter is to provide a data fusion (DF) scheme for combining in a unique view, radar and visual data. The experimental evaluation of the performance for the modules included in the framework has been carried out using publicly available data from the VOC dataset and the MarDT - Maritime Detection and Tracking (MarDT) data set, containing data coming from different real VTS systems, with ground truth information. Moreover, an operative scenario where traditional VTS systems can benefit from the proposed approach is presented.
---
paper_title: Model-based segmentation of FLIR images

paper_content:

The use of gray-scale intensities together with the edge information present in a forward-looking infrared (FLIR) image to obtain a precise and accurate segmentation of a target is presented. A model of FLIR images based on gray-scale and edge information is incorporated in a gradient relaxation technique which explicitly maximizes a criterion function based on the inconsistency and ambiguity of classification of pixels with respect to their neighbors. Four variations of the basic technique which provide automatic selection of thresholds to segment FLIR images are considered. These methods are compared, and several examples of segmentation of ship images are given. >
---
paper_title: Detection and classification of infrared decoys and small targets in a sea background

paper_content:

A combination of algorithms has been developed for the detection, tracking, and classification of targets at sea. In a flexible software setup, different methods of preprocessing and detection can be chosen for the processing of infrared and visible-light images. Two projects, in which the software is used, are discussed. In the SURFER project, the algorithms are used for the detection and classification of small targets, e.g., swimmers, dinghies, speedboats, and floating mines. Different detection methods are applied to recorded data. We will present a method to describe the background by fitting continuous functions to the data, and show that this provides a better separation between objects and clutter. The detection of targets using electro- optical systems is one part of this project, in which also algorithms for fusion of electro-optical data with radar data are being developed. In the second project, a simple infrared image-seeker has been built that is used to test the effectiveness of infrared decoys launched from a ship. In a more complicated image seeker algorithm, features such as contrast and size and characterization of trajectory are used to differentiate between ship, infrared decoys and false alarms resulting from clutter. In this paper, results for the detection of small targets in a sea background are shown for a number of detection methods. Further, a description is given of the simulator imaging seeker, and some results of the imaging seeker software applied to simulated and recorded data will be shown.
---
paper_title: MuSCoWERT: multi-scale consistence of weighted edge Radon transform for horizon detection in maritime images.

paper_content:

This paper addresses the problem of horizon detection, a fundamental process in numerous object detection algorithms, in a maritime environment. The maritime environment is characterized by the absence of fixed features, the presence of numerous linear features in dynamically changing objects and background and constantly varying illumination, rendering the typically simple problem of detecting the horizon a challenging one. We present a novel method called multi-scale consistence of weighted edge Radon transform, abbreviated as MuSCoWERT. It detects the long linear features consistent over multiple scales using multi-scale median filtering of the image followed by Radon transform on a weighted edge map and computing the histogram of the detected linear features. We show that MuSCoWERT has excellent performance, better than seven other contemporary methods, for 84 challenging maritime videos, containing over 33,000 frames, and captured using visible range and near-infrared range sensors mounted onboard, onshore, or on floating buoys. It has a median error of about 2 pixels (less than 0.2%) from the center of the actual horizon and a median angular error of less than 0.4 deg. We are also sharing a new challenging horizon detection dataset of 65 videos of visible, infrared cameras for onshore and onboard ship camera placement.
---
paper_title: Clutter-adaptive infrared small target detection in infrared maritime scenarios

paper_content:

A clutter-adaptive target-detection method is proposed to detect small moving targets in midwave infrared maritime scenarios. In particular, we focus our attention on the sea-sky background and targets of interest are ships along the horizontal sea-sky line. In the distant sea-sky background infrared imagery, small targets frequently appear as weak intensity and do not have enough structure information in the vicinity of horizontal sea-sky-line, that is sea-sky region, and the complexity of background clutters has direct impact on the target detection performance. Thus, a fuzzy system constructed by the extracted image-based features is designed for the clutter classification. And then, based on the determined clutter type, low clutter or high clutter, the horizontal sea-sky line can be detected successfully. In the target detection stage, under the guidance of the sea-sky line, a modified multilevel filter method is applied to enhance the target in the sea-sky region. Finally, the recursive segmentation method and the target validation technique are adopted for the target extraction and validation. In the experiments, the target detection performance of proposed method is validated by extensive experiments on infrared images taken in different imaging conditions. It achieves high accuracy with a low false alarm rate.
---
paper_title: Infrared small target enhancement by using sequential top-hat filters

paper_content:

Generally in the infrared images, the targets have low contrast with the background, which makes the detection of the small targets difficult. To improve the detectability of the infrared small targets, this paper presents a novel algorithm for infrared small target enhancement by using sequential top-hat filters. Moreover, the proposed algorithm has been compared with several existing algorithms. The experimental results indicate that sequential top-hat filters could well enhance the infrared small targets and effectively suppress the background clutters.
---
paper_title: Horizon Detection Using Machine Learning Techniques

paper_content:

Detecting a horizon in an image is an important part of many image related applications such as detecting ships on the horizon, flight control, and port security. Most of the existing solutions for the problem only use image processing methods to identify a horizon line in an image. This results in good accuracy for many cases and is fast in computation. However, for some images with difficult environmental conditions like a foggy or cloudy sky these image processing methods are inherently inaccurate in identifying the correct horizon. This paper investigates how to detect the horizon line in a set of images using a machine learning approach. The performance of the SVM, J48, and Na?ve Bayes classifiers, used for the problem, has been compared. Accuracy of 90-99% in identifying horizon was achieved on image data set of 20 images.
---
paper_title: Polynomial background estimation using visible light video streams for robust automatic detection in a maritime environment

paper_content:

For naval surveillance, automatic detection of surface objects, like vessels, in a maritime environment is an ::: important contribution of the Electro-Optical (EO) sensor systems on board. Based on previous research using ::: single images, a background estimation approach using low-order polynomials is proposed for the automatic ::: detection of objects in a maritime environment. The polynomials are fitted to the intensity values in the image ::: after which the deviation between the fitted intensity values and the measured intensity values are used for detection. The research presented in this paper, includes the time information by using video streams instead of single images. Hereby, the level of fusing time information and the number of frames necessary for stable detection and tracking behaviour are analysed and discussed. The performance of the detection approach is tested on a, during the fall of 2007, collected extensive dataset of maritime pictures in the Mediterranean Sea and in the North Sea on board of an Air Defence Command frigate, HNLMS Tromp.
---
paper_title: MSCM-LiFe: Multi-scale cross modal linear feature for horizon detection in maritime images

paper_content:

This paper proposes a new method for horizon detection called the multi-scale cross modal linear feature. This method integrates three different concepts related to the presence of horizon in maritime images to increase the accuracy of horizon detection. Specifically it uses the persistence of horizon in multi-scale median filtering, and its detection as a linear feature commonly detected by two different methods, namely the Hough transform of edgemap and the intensity gradient. We demonstrate the performance of the method over 13 videos comprising of more than 3000 frames and show that the proposed method detects horizon with small error in most of the cases, outperforming three state-of-the-art methods.
---
paper_title: Vision-guided flight stability and control for micro air vehicles

paper_content:

Substantial progress has been made recently towards designing, building and test-flying remotely piloted micro air vehicles (MAVs) and small unmanned air vehicles. We seek to complement this progress in overcoming the aerodynamic obstacles to flight at very small scales with a visionguided flight stability and autonomy system, based on a robust horizon detection algorithm. In this paper, we first motivate the use of computer vision for MAV autonomy, arguing that given current sensor technology, vision may be the only practical approach to the problem. We then describe our statistical vision-based horizon detection algorithm, which has been demonstrated at 30 Hz with over 99.9% correct horizon identification. Next, we develop robust schemes for the detection of extreme MAV attitudes, where no horizon is visible, and for the detection of horizon estimation errors, due to external factors such as video transmission noise. Finally, we discuss our feedback controller for selfstabilized flight and report result...
---
paper_title: A vision system for intelligent mission profiles of micro air vehicles

paper_content:

Recently, much progress has been made toward the development of small-scale aircraft, known broadly as Micro Air Vehicles (MAVs). Until recently, these platforms were exclusively remotely piloted, with no autonomous or intelligent capabilities, due at least in part to stringent payload restrictions that limit onboard sensors. However, the one sensor that is critical to most conceivable MAV missions, such as remote surveillance, is an onboard video camera and transmitter that streams flight video to a nearby ground station. Exploitation of this key sensor is, therefore, desirable, since no additional onboard hardware (and weight) is required. As such, in this paper we develop a general and unified computer vision framework for MAVs that not only addresses basic flight stability and control, but enables more intelligent missions as well. This paper is organized as follows. We first develop a real-time feature extraction method called multiscale linear discriminant analysis (MLDA), which explicitly incorporates color into its feature representation, while implicitly encoding texture through a dynamic multiscale representation of image details. We demonstrate key advantages of MLDA over other possible multiscale approaches (e.g., wavelets), especially in dealing with transient video noise. Next, we show that MLDA provides a natural framework for performing real-time horizon detection. We report horizon-detection results for a range of images differing in lighting and scenery and quantify performance as a function of image noise. Furthermore, we show how horizon detection naturally leads to closed-loop flight stabilization. Then, we motivate the use of tree-structured belief networks (TSBNs) with MLDA features for sky/ground segmentation. This type of segmentation augments basic horizon detection and enables certain MAV missions where prior assumptions about the flight vehicle's orientation are not possible. Again, we report segmentation results for a range of images and quantify robustness to image noise. Finally, we demonstrate the seamless extension of this framework, through the idea of visual contexts, for the detection of artificial objects and/or structures and illustrate several examples of such additional segmentation. This extension thus enables mission profiles that require, for example, following a specific road or the tracking of moving ground objects. Throughout, our approach and algorithms are heavily influenced by real-time constraints and robustness to transient video noise.
---
paper_title: Detection and tracking of ships in open sea with rapidly moving buoy-mounted camera system

paper_content:

Abstract Visual surveillance in the maritime domain has been explored for more than a decade. Although it has produced a number of working systems and resulted in a mature technology, surveillance has been restricted to the port facilities or areas close to the coast line assuming a fixed-camera scenario. This paper presents a novel algorithm for open-sea visual maritime surveillance. We explore a challenging situation with a forward-looking camera mounted on a buoy or other floating platform. The proposed algorithm detects, localizes, and tracks ships in the field of view of the camera. Specifically, developed algorithm is uniquely designed to handle rapidly moving camera. Its performance is robust in the presence of a random relatively-large camera motion. In the context of ship detection we developed a new horizon detection scheme for a complex maritime domain. The performance of our algorithm and its comprising elements is evaluated. Ship detection precision of 88% is achieved on a large dataset collected from a prototype system.
---
paper_title: Omnidirectional vision on UAV for attitude computation

paper_content:

Unmanned aerial vehicles (UAVs) are the subject of an increasing interest in many applications. Autonomy is one of the major advantages of these vehicles. It is then necessary to develop particular sensors in order to provide efficient navigation functions. In this paper, we propose a method for attitude computation catadioptric images. We first demonstrate the advantages of the catadioptric vision sensor for this application. In fact, the geometric properties of the sensor permit to compute easily the roll and pitch angles. The method consists in separating the sky from the earth in order to detect the horizon. We propose an adaptation of the Markov random fields for catadioptric images for this segmentation. The second step consists in estimating the parameters of the horizon line thanks to a robust estimation algorithm. We also present the angle estimation algorithm and finally, we show experimental results on synthetic and real images captured from an airplane
---
paper_title: Vision-based horizon extraction for micro air vehicle flight control

paper_content:

Recently, more and more research has been done on micro air vehicles (MAVs). An autonomous flight control system is necessary for developing practical MAVs to be used for a wide array of missions. Due to the limitations of size, weight, and power, MAVs have the very low payload capacity and moments of inertia. The current technologies with rate and acceleration sensors applied on larger aircrafts are impractical to MAVs, and they are difficult to be scaled down to satisfy the demands of MAVs. Since surveillance has been considered as the primary mission of MAVs, it is essential for MAVs to be equipped with on-board imaging sensors such as cameras, which have rich information content. So vision-based techniques, without increasing the MAVs payload, may be a feasible idea for flight autonomy of MAVs. In this paper, a new robust horizon extraction algorithm based on the orientation projection method is proposed, which is the foundation of a vision-based flight control system. The horizon extraction algorithm is effective for both color images and gray images. The horizon can be extracted not only from fine images captured in fair conditions but also from blurred images captured in cloudy, even foggy days. In order to raise the computational speed to meet real-time requirements, the algorithmic optimization is also discussed in the paper, which is timesaving by narrowing the seeking scope of orientations and adopting the table look-up method. According to the orientation and position of the horizon in the image, two important angular attitude parameters for stability and control, the roll angle and the pitch angle, could be calculated. Several experimental results demonstrate the feasibility and robustness of the algorithm.
---
paper_title: Automatic detection of small surface targets with electro-optical sensors in a harbor environment

paper_content:

In modern warfare scenarios naval ships must operate in coastal environments. These complex environments, in bays and narrow straits, with cluttered littoral backgrounds and many civilian ships may contain asymmetric threats of fast targets, such as rhibs, cabin boats and jet-skis. Optical sensors, in combination with image enhancement and automatic detection, assist an operator to reduce the response time, which is crucial for the protection of the naval and land-based supporting forces. In this paper, we present our work on automatic detection of small surface targets which includes multi-scale horizon detection and robust estimation of the background intensity. To evaluate the performance of our detection technology, data was recorded with both infrared and visual-light cameras in a coastal zone and in a harbor environment. During these trials multiple small targets were used. Results of this evaluation are shown in this paper.
---
paper_title: Real-world multisensor image alignment using edge focusing and Hausdorff distances

paper_content:

The area-based methods, such as using Laplacian pyramid and Fourier transform-based phase matching, benefit by highlighting high spatial frequencies to reduce sensitivity to the feature inconsistency problem in the multisensor image registration. The feature extraction and matching methods are more powerful and versatile to process poor quality IR images. We implement multi-scale hierarchical edge detection and edge focusing and introduce a new salience measure for the horizon, for multisensor image registration. The common features extracted from images of two modalities can be still different in detail. Therefore, the transformation space match methods with the Hausdorff distance measure is more suitable than the direct feature matching methods. We have introduced image quadtree partition technique to the Hausdorff distance matching, that dramatically reduces the size of the search space. Image registration of real world visible/IR images of battle fields is shown.
---
paper_title: Detection and tracking of ships in open sea with rapidly moving buoy-mounted camera system

paper_content:

Abstract Visual surveillance in the maritime domain has been explored for more than a decade. Although it has produced a number of working systems and resulted in a mature technology, surveillance has been restricted to the port facilities or areas close to the coast line assuming a fixed-camera scenario. This paper presents a novel algorithm for open-sea visual maritime surveillance. We explore a challenging situation with a forward-looking camera mounted on a buoy or other floating platform. The proposed algorithm detects, localizes, and tracks ships in the field of view of the camera. Specifically, developed algorithm is uniquely designed to handle rapidly moving camera. Its performance is robust in the presence of a random relatively-large camera motion. In the context of ship detection we developed a new horizon detection scheme for a complex maritime domain. The performance of our algorithm and its comprising elements is evaluated. Ship detection precision of 88% is achieved on a large dataset collected from a prototype system.
---
paper_title: Automatic detection of small surface targets with electro-optical sensors in a harbor environment

paper_content:

In modern warfare scenarios naval ships must operate in coastal environments. These complex environments, in bays and narrow straits, with cluttered littoral backgrounds and many civilian ships may contain asymmetric threats of fast targets, such as rhibs, cabin boats and jet-skis. Optical sensors, in combination with image enhancement and automatic detection, assist an operator to reduce the response time, which is crucial for the protection of the naval and land-based supporting forces. In this paper, we present our work on automatic detection of small surface targets which includes multi-scale horizon detection and robust estimation of the background intensity. To evaluate the performance of our detection technology, data was recorded with both infrared and visual-light cameras in a coastal zone and in a harbor environment. During these trials multiple small targets were used. Results of this evaluation are shown in this paper.
---
paper_title: Model-based segmentation of FLIR images

paper_content:

The use of gray-scale intensities together with the edge information present in a forward-looking infrared (FLIR) image to obtain a precise and accurate segmentation of a target is presented. A model of FLIR images based on gray-scale and edge information is incorporated in a gradient relaxation technique which explicitly maximizes a criterion function based on the inconsistency and ambiguity of classification of pixels with respect to their neighbors. Four variations of the basic technique which provide automatic selection of thresholds to segment FLIR images are considered. These methods are compared, and several examples of segmentation of ship images are given. >
---
paper_title: Detection and classification of infrared decoys and small targets in a sea background

paper_content:

A combination of algorithms has been developed for the detection, tracking, and classification of targets at sea. In a flexible software setup, different methods of preprocessing and detection can be chosen for the processing of infrared and visible-light images. Two projects, in which the software is used, are discussed. In the SURFER project, the algorithms are used for the detection and classification of small targets, e.g., swimmers, dinghies, speedboats, and floating mines. Different detection methods are applied to recorded data. We will present a method to describe the background by fitting continuous functions to the data, and show that this provides a better separation between objects and clutter. The detection of targets using electro- optical systems is one part of this project, in which also algorithms for fusion of electro-optical data with radar data are being developed. In the second project, a simple infrared image-seeker has been built that is used to test the effectiveness of infrared decoys launched from a ship. In a more complicated image seeker algorithm, features such as contrast and size and characterization of trajectory are used to differentiate between ship, infrared decoys and false alarms resulting from clutter. In this paper, results for the detection of small targets in a sea background are shown for a number of detection methods. Further, a description is given of the simulator imaging seeker, and some results of the imaging seeker software applied to simulated and recorded data will be shown.
---
paper_title: Using histograms to detect and track objects in color video

paper_content:

Two methods of detecting and tracking objects in color video are presented. Color and edge histograms are explored as ways to model the background and foreground of a scene. The two types of methods are evaluated to determine their speed, accuracy and robustness. Histogram comparison techniques are used to compute similarity values that aid in identifying regions of interest. Foreground objects are detected and tracked by dividing each video frame into smaller regions (cells) and comparing the histogram of each cell to the background model. Results are presented for video sequences of human activity.
---
paper_title: Target detection of maritime search and rescue: Saliency accumulation method

paper_content:

Target detection using visual image sequence is an important task of maritime search and rescue. A novel saliency accumulation approach is proposed. Each frame of image sequence is firstly transformed to LAB space. The saliency maps for two color channels and intensity channel are obtained by spectral residual in frequency domain and the local saliency in space domain taking advantages of both approaches. Then saliency map of every frame is fusion of those from each weighted feature. In order to reduce the sea clutters or detect the weak small target, the saliency maps from successive frames are accumulated to reach a binary saliency map by applying a threshold. The approach is simple and effective. Experiments are conducted to demonstrate the validity of proposed approach for target detection in maritime search and rescue.
---
paper_title: Aquatic debris monitoring using smartphone-based robotic sensors

paper_content:

Monitoring aquatic debris is of great interest to the ecosystems, marine life, human health, and water transport. This paper presents the design and implementation of SOAR - a vision-based surveillance robot system that integrates an off-the-shelf Android smartphone and a gliding robotic fish for debris monitoring. SOAR features real-time debris detection and coverage-based rotation scheduling algorithms. The image processing algorithms for debris detection are specifically designed to address the unique challenges in aquatic environments. The rotation scheduling algorithm provides effective coverage of sporadic debris arrivals despite camera's limited angular view. Moreover, SOAR is able to dynamically offload computation-intensive processing tasks to the cloud for battery power conservation. We have implemented a SOAR prototype and conducted extensive experimental evaluation. The results show that SOAR can accurately detect debris in the presence of various environment and system dynamics, and the rotation scheduling algorithm enables SOAR to capture debris arrivals with reduced energy consumption.
---
paper_title: Pfinder: Real-time tracking of the human body

paper_content:

Pfinder is a real-time system for tracking people and interpreting their behavior. It runs at 10 Hz on a standard SGI Indy computer, and has performed reliably on thousands of people in many different physical locations. The system uses a multiclass statistical model of color and shape to obtain a 2D representation of head and hands in a wide range of viewing conditions. Pfinder has been successfully used in a wide range of applications including wireless interfaces, video databases, and low-bandwidth coding.
---
paper_title: A Hybrid Color-Based Foreground Object Detection Method for Automated Marine Surveillance

paper_content:

This paper proposes a hybrid foreground object detection method suitable for the marine surveillance applications. Our approach combines an existing foreground object detection method with an image color segmentation technique to improve accuracy. The foreground segmentation method employs a Bayesian decision framework, while the color segmentation part is graph-based and relies on the local variation of edges. We also establish the set of requirements any practical marine surveillance algorithm should fulfill, and show that our method conforms to these requirements. Experiments show good results in the domain of marine surveillance sequences.
---
paper_title: Saliency Detection: A Spectral Residual Approach

paper_content:

The ability of human visual system to detect visual saliency is extraordinarily fast and reliable. However, computational modeling of this basic intelligent behavior still remains a challenge. This paper presents a simple method for the visual saliency detection. Our model is independent of features, categories, or other forms of prior knowledge of the objects. By analyzing the log-spectrum of an input image, we extract the spectral residual of an image in spectral domain, and propose a fast method to construct the corresponding saliency map in spatial domain. We test this model on both natural pictures and artificial images such as psychological patterns. The result indicate fast and robust saliency detection of our method.
---
paper_title: Adaptive background mixture models for real-time tracking

paper_content:

A common method for real-time segmentation of moving regions in image sequences involves "background subtraction", or thresholding the error between an estimate of the image without moving objects and the current image. The numerous approaches to this problem differ in the type of background model used and the procedure used to update the model. This paper discusses modeling each pixel as a mixture of Gaussians and using an on-line approximation to update the model. The Gaussian, distributions of the adaptive mixture model are then evaluated to determine which are most likely to result from a background process. Each pixel is classified based on whether the Gaussian distribution which represents it most effectively is considered part of the background model. This results in a stable, real-time outdoor tracker which reliably deals with lighting changes, repetitive motions from clutter, and long-term scene changes. This system has been run almost continuously for 16 months, 24 hours a day, through rain and snow.
---
paper_title: Spatio-temporal Saliency detection using phase spectrum of quaternion fourier transform

paper_content:

Salient areas in natural scenes are generally regarded as the candidates of attention focus in human eyes, which is the key stage in object detection. In computer vision, many models have been proposed to simulate the behavior of eyes such as SaliencyToolBox (STB), neuromorphic vision toolkit (NVT) and etc., but they demand high computational cost and their remarkable results mostly rely on the choice of parameters. Recently a simple and fast approach based on Fourier transform called spectral residual (SR) was proposed, which used SR of the amplitude spectrum to obtain the saliency map. The results are good, but the reason is questionable.
---
paper_title: Robust Computer Vision: An Interdisciplinary Challenge

paper_content:

This special issue is dedicated to examining the use of techniques from robust statistics in solving computer vision problems. It represents a milestone of recent progress within a subarea of our field that is nearly as old as the field itself, but has seen rapid growth over the past decade. Our Introduction considers the meaning of robustness in computer vision, summarizes the papers, and outlines the relationship between techniques in computer vision and statistics as a means of highlighting future directions. It complements the available reviews on this topic [12, 13].
---
paper_title: Detection and tracking of moving objects in a maritime environment using level set with shape priors

paper_content:

Over the years, maritime surveillance has become increasingly important due to the recurrence of piracy. While surveillance has traditionally been a manual task using crew members in lookout positions on parts of the ship, much work is being done to automate this task using digital cameras coupled with a computer that uses image processing techniques that intelligently track object in the maritime environment. One such technique is level set segmentation which evolves a contour to objects of interest in a given image. This method works well but gives incorrect segmentation results when a target object is corrupted in the image. This paper explores the possibility of factoring in prior knowledge of a ship’s shape into level set segmentation to improve results, a concept that is unaddressed in maritime surveillance problem. It is shown that the developed video tracking system outperforms level set-based systems that do not use prior shape knowledge, working well even where these systems fail.
---
paper_title: Real-time automatic small infrared target detection using local spectral filtering in the frequency

paper_content:

Accurate and fast detection of small infrared target has very important meaning for infrared precise guidance, early ::: warning, video surveillance, etc. Based on human visual attention mechanism, an automatic detection algorithm for ::: small infrared target is presented. In this paper, instead of searching for infrared targets, we model regular patches that do ::: not attract much attention by our visual system. This is inspired by the property that the regular patches in spatial domain ::: turn out to correspond to the spikes in the amplitude spectrum. Unlike recent approaches using global spectral filtering, ::: we define the concept of local maxima suppression using local spectral filtering to smooth the spikes in the amplitude ::: spectrum, thereby producing the pop-out of the infrared targets. In the proposed method, we firstly compute the ::: amplitude spectrum of an input infrared image. Second, we find the local maxima of the amplitude spectrum using cubic ::: facet model. Third, we suppress the local maxima using the convolution of the local spectrum with a low-pass Gaussian ::: kernel of an appropriate scale. At last, the detection result in spatial domain is obtained by reconstructing the 2D signal ::: using the original phase and the log amplitude spectrum by suppressing local maxima. The experiments are performed ::: for some real-life IR images, and the results prove that the proposed method has satisfying detection effectiveness and ::: robustness. Meanwhile, it has high detection efficiency and can be further used for real-time detection and tracking.
---
paper_title: Argos - a Video Surveillance System for boat Traffic Monitoring in Venice

paper_content:

Visual surveillance in dynamic scenes is currently one of the most active research topics in computer vision, many existing applications are available. However, difficulties in realizing effective video surveillance systems that are robust to the many different conditions that arise in real environments, make the actual deployment of such systems very challenging. In this article, we present a real, unique and pioneer video surveillance system for boat traffic monitoring, ARGOS. The system runs continuously 24 hours a day, 7 days a week, day and night in the city of Venice (Italy) since 2007 and it is able to build a reliable background model of the water channel and to track the boats navigating the channel with good accuracy in real-time. A significant experimental evaluation, reported in this article, has been performed in order to assess the real performance of the system.
---
paper_title: Multispectral Target Detection and Tracking for Seaport Video Surveillance

paper_content:

In this paper, a video surveillance process is presented including target detection and tracking of ships at the entrance of a seaport in order to improve security and to prevent terrorist attacks. This process is helpful in the automatic analysis of movements inside the seaport. Steps of detection and tracking are completed using IR data whereas the pattern recognition stage is achieved on color data. A comparison of results of detection and tracking is presented on both IR and color data in order to justify the choice of IR images for these two steps. A draft description of the pattern recognition stage is finally drawn up as development prospect.
---
paper_title: Clutter-adaptive infrared small target detection in infrared maritime scenarios

paper_content:

A clutter-adaptive target-detection method is proposed to detect small moving targets in midwave infrared maritime scenarios. In particular, we focus our attention on the sea-sky background and targets of interest are ships along the horizontal sea-sky line. In the distant sea-sky background infrared imagery, small targets frequently appear as weak intensity and do not have enough structure information in the vicinity of horizontal sea-sky-line, that is sea-sky region, and the complexity of background clutters has direct impact on the target detection performance. Thus, a fuzzy system constructed by the extracted image-based features is designed for the clutter classification. And then, based on the determined clutter type, low clutter or high clutter, the horizontal sea-sky line can be detected successfully. In the target detection stage, under the guidance of the sea-sky line, a modified multilevel filter method is applied to enhance the target in the sea-sky region. Finally, the recursive segmentation method and the target validation technique are adopted for the target extraction and validation. In the experiments, the target detection performance of proposed method is validated by extensive experiments on infrared images taken in different imaging conditions. It achieves high accuracy with a low false alarm rate.
---
paper_title: Infrared small target enhancement by using sequential top-hat filters

paper_content:

Generally in the infrared images, the targets have low contrast with the background, which makes the detection of the small targets difficult. To improve the detectability of the infrared small targets, this paper presents a novel algorithm for infrared small target enhancement by using sequential top-hat filters. Moreover, the proposed algorithm has been compared with several existing algorithms. The experimental results indicate that sequential top-hat filters could well enhance the infrared small targets and effectively suppress the background clutters.
---
paper_title: Foreground object detection from videos containing complex background

paper_content:

This paper proposes a novel method for detection and segmentation of foreground objects from a video which contains both stationary and moving background objects and undergoes both gradual and sudden "once-off" changes. A Bayes decision rule for classification of background and foreground from selected feature vectors is formulated. Under this rule, different types of background objects will be classified from foreground objects by choosing a proper feature vector. The stationary background object is described by the color feature, and the moving background object is represented by the color co-occurrence feature. Foreground objects are extracted by fusing the classification results from both stationary and moving pixels. Learning strategies for the gradual and sudden "once-off" background changes are proposed to adapt to various changes in background through the video. The convergence of the learning process is proved and a formula to select a proper learning rate is also derived. Experiments have shown promising results in extracting foreground objects from many complex backgrounds including wavering tree branches, flickering screens and water surfaces, moving escalators, opening and closing doors, switching lights and shadows of moving objects.
---
paper_title: Horizon Detection Using Machine Learning Techniques

paper_content:

Detecting a horizon in an image is an important part of many image related applications such as detecting ships on the horizon, flight control, and port security. Most of the existing solutions for the problem only use image processing methods to identify a horizon line in an image. This results in good accuracy for many cases and is fast in computation. However, for some images with difficult environmental conditions like a foggy or cloudy sky these image processing methods are inherently inaccurate in identifying the correct horizon. This paper investigates how to detect the horizon line in a set of images using a machine learning approach. The performance of the SVM, J48, and Na?ve Bayes classifiers, used for the problem, has been compared. Accuracy of 90-99% in identifying horizon was achieved on image data set of 20 images.
---
paper_title: Small target detection under the sea using multi-scale spectral residual and maximum symmetric surround

paper_content:

Detection of small salient targets under the sea is useful for maritime visual surveillance, maritime search and rescue, and ship collision avoidance. In this paper, we propose a small target detection method based on visual saliency map, which is generated by the multi-scale spectral residual and maximum symmetric surround methods. First, the multi-scale spectral residual method is used to detect the targets of different sizes. Then the maximum symmetric surround method is employed to improve the resolution and suppress background for each scale. The final saliency map is obtained by finding the best scale using the 2D entropy. We compare our approach to three salient region detection methods with ground truth and a salient target segmentation application. The experimental results show that the proposed method outperforms the other three methods in both qualitative and quantitative terms.
---
paper_title: Detection and location of people in video images using adaptive fusion of color and edge information

paper_content:

A new method of finding people in video images is presented. The detection is based on a novel background modeling and subtraction approach which uses both color and edge information. We introduce confidence maps gray-scale images whose intensity is a function of confidence that a pixel has changed - to fuse intermediate results and represent the results of background subtraction. The latter is used to delineate a person's body by guiding contour collection to segment the person from the background. The method is tolerant to scene clutter, slow illumination changes, and camera noise, and runs in near real time on a standard platform.
---
paper_title: Identification and tracking of maritime objects in near-infrared image sequences for collision avoidance

paper_content:

This paper describes the continuing development of an image processing system for use on high-speed passenger ferries. The system automatically identifies objects in a maritime scene and uses the detected motion to alert a human observer to potential collision situations. Three integrated image-processing algorithms, namely an image preprocessor, a motion cue generator, and a target tracker, perform the identification and tracking of maritime objects. The pre-processing filters the image and applies a histogram technique to segment the sea from potential objects of interest. The segmented image is passed to the motion cue generator, which provides motion cues based on the differences between consecutive frames of segmented image data. The target tracker applies dynamic constraints on object motion to solve the correspondence problem, thus increasing the confidence that an identified object is a target. Identified and tracked objects are highlighted to a human observer using a white box viewing cue placed directly around the object of interest.
---
paper_title: Detection and tracking of ships in open sea with rapidly moving buoy-mounted camera system

paper_content:

Abstract Visual surveillance in the maritime domain has been explored for more than a decade. Although it has produced a number of working systems and resulted in a mature technology, surveillance has been restricted to the port facilities or areas close to the coast line assuming a fixed-camera scenario. This paper presents a novel algorithm for open-sea visual maritime surveillance. We explore a challenging situation with a forward-looking camera mounted on a buoy or other floating platform. The proposed algorithm detects, localizes, and tracks ships in the field of view of the camera. Specifically, developed algorithm is uniquely designed to handle rapidly moving camera. Its performance is robust in the presence of a random relatively-large camera motion. In the context of ship detection we developed a new horizon detection scheme for a complex maritime domain. The performance of our algorithm and its comprising elements is evaluated. Ship detection precision of 88% is achieved on a large dataset collected from a prototype system.
---
paper_title: Adaptive maritime video surveillance

paper_content:

Maritime assets such as ports, harbors, and vessels are vulnerable to a variety of near-shore threats such as small-boat attacks. Currently, such vulnerabilities are addressed predominantly by watchstanders and manual video surveillance, which is manpower intensive. Automatic maritime video surveillance techniques are being introduced to reduce manpower costs, but they have limited functionality and performance. For example, they only detect simple events such as perimeter breaches and cannot predict emerging threats. They also generate too many false alerts and cannot explain their reasoning. To overcome these limitations, we are developing the Maritime Activity Analysis Workbench (MAAW), which will be a mixed-initiative real-time maritime video surveillance tool that uses an integrated supervised machine learning approach to label independent and coordinated maritime activities. It uses the same information to predict anomalous behavior and explain its reasoning; this is an important capability for watchstander training and for collecting performance feedback. In this paper, we describe MAAW's functional architecture, which includes the following pipeline of components: (1) a video acquisition and preprocessing component that detects and tracks vessels in video images, (2) a vessel categorization and activity labeling component that uses standard and relational supervised machine learning methods to label maritime activities, and (3) an ontology-guided vessel and maritime activity annotator to enable subject matter experts (e.g., watchstanders) to provide feedback and supervision to the system. We report our findings from a preliminary system evaluation on river traffic video.
---
paper_title: Traditional and Recent Approaches in Background Modeling for Foreground Detection: An Overview

paper_content:

Background modeling for foreground detection is often used in different applications ::: to model the background and then detect the moving objects in the scene like in video ::: surveillance. The last decade witnessed very significant publications in this field. Furthermore, ::: several surveys can be found in literature but none of them addresses an overall ::: review in this field. So, the purpose of this paper is to provide a complete survey ::: of the traditional and recent approaches. First, we categorize the different approaches ::: found in literature. We have classified them in terms of the mathematical models used ::: and we have discussed them in terms of the critical situations that they claim to handle. ::: Furthermore, we present the available resources, datasets and libraries. Then, we ::: conclude with several promising directions for future research.
---
paper_title: Automatic detection of small surface targets with electro-optical sensors in a harbor environment

paper_content:

In modern warfare scenarios naval ships must operate in coastal environments. These complex environments, in bays and narrow straits, with cluttered littoral backgrounds and many civilian ships may contain asymmetric threats of fast targets, such as rhibs, cabin boats and jet-skis. Optical sensors, in combination with image enhancement and automatic detection, assist an operator to reduce the response time, which is crucial for the protection of the naval and land-based supporting forces. In this paper, we present our work on automatic detection of small surface targets which includes multi-scale horizon detection and robust estimation of the background intensity. To evaluate the performance of our detection technology, data was recorded with both infrared and visual-light cameras in a coastal zone and in a harbor environment. During these trials multiple small targets were used. Results of this evaluation are shown in this paper.
---
paper_title: Tracking Ships from Fast Moving Camera through Image Registration

paper_content:

This paper presents an algorithm that detects and tracks marine vessels in video taken by a nonstationary camera installed on an untethered buoy. The video is characterized by large inter-frame motion of the camera, cluttered background, and presence of compression artifacts. Our approach performs segmentation of ships in individual frames processed with a color-gradient filter. The threshold selection is based on the histogram of the search region. Tracking of ships in a sequence is enabled by registering the horizon images in one coordinate system and by using a multihypothesis framework. Registration step uses an area-based technique to correlate a processed strip of the image over the found horizon line. The results of evaluation of detection, localization, and tracking of the ships show significant increase in performance in comparison to the previously used technique.
---
paper_title: Automatic Obstacle Detection for USV’s Navigation Using Vision Sensors

paper_content:

This paper presents an automatic method to acquire, identify, and track obstacles from an Unmanned Surface Vehicle (USV) location in marine environments using 2D Commercial Of The Shelf (COTS) video sensors, and analyzing video streams as input. The guiding line of this research is to develop real-time automatic identification and tracking abilities in marine environment with COTS sensors. The output of this algorithm provides obstacle’s location in x-y coordinates. The ability to recognize and identify obstacles becomes more essential for USV’s autonomous capabilities, such as obstacle avoidance, decision modules, and other Artificial Intelligence (AI) abilities using low cost sensors. Our algorithm is not based on obstacles characterization. Algorithm performances were tested in various scenarios with real-time USV’s video streams, indicating that the algorithm can be used for real-time applications with high success rate and fast time computation.
---
paper_title: A Novel Hierarchical Method of Ship Detection from Spaceborne Optical Image Based on Shape and Texture Features

paper_content:

Ship detection from remote sensing imagery is very important, with a wide array of applications in areas such as fishery management, vessel traffic services, and naval warfare. This paper focuses on the issue of ship detection from spaceborne optical images (SDSOI). Although advantages of synthetic-aperture radar (SAR) result in that most of current ship detection approaches are based on SAR images, disadvantages of SAR still exist, such as the limited number of SAR sensors, the relatively long revisit cycle, and the relatively lower resolution. With the increasing number of and the resulting improvement in continuous coverage of the optical sensors, SDSOI can partly overcome the shortcomings of SAR-based approaches and should be investigated to help satisfy the requirements of real-time ship monitoring. In SDSOI, several factors such as clouds, ocean waves, and small islands affect the performance of ship detection. This paper proposes a novel hierarchical complete and operational SDSOI approach based on shape and texture features, which is considered a sequential coarse-to-fine elimination process of false alarms. First, simple shape analysis is adopted to eliminate evident false candidates generated by image segmentation with global and local information and to extract ship candidates with missing alarms as low as possible. Second, a novel semisupervised hierarchical classification approach based on various features is presented to distinguish between ships and nonships to remove most false alarms. Besides a complete and operational SDSOI approach, the other contributions of our approach include the following three aspects: 1) it classifies ship candidates by using their class probability distributions rather than the direct extracted features; 2) the relevant classes are automatically built by the samples' appearances and their feature attribute in a semisupervised mode; and 3) besides commonly used shape and texture features, a new texture operator, i.e., local multiple patterns, is introduced to enhance the representation ability of the feature set in feature extraction. Experimental results of SDSOI on a large image set captured by optical sensors from multiple satellites show that our approach is effective in distinguishing between ships and nonships, and obtains a satisfactory ship detection performance.
---
paper_title: Model-based segmentation of FLIR images

paper_content:

The use of gray-scale intensities together with the edge information present in a forward-looking infrared (FLIR) image to obtain a precise and accurate segmentation of a target is presented. A model of FLIR images based on gray-scale and edge information is incorporated in a gradient relaxation technique which explicitly maximizes a criterion function based on the inconsistency and ambiguity of classification of pixels with respect to their neighbors. Four variations of the basic technique which provide automatic selection of thresholds to segment FLIR images are considered. These methods are compared, and several examples of segmentation of ship images are given. >
---
paper_title: A Hybrid Color-Based Foreground Object Detection Method for Automated Marine Surveillance

paper_content:

This paper proposes a hybrid foreground object detection method suitable for the marine surveillance applications. Our approach combines an existing foreground object detection method with an image color segmentation technique to improve accuracy. The foreground segmentation method employs a Bayesian decision framework, while the color segmentation part is graph-based and relies on the local variation of edges. We also establish the set of requirements any practical marine surveillance algorithm should fulfill, and show that our method conforms to these requirements. Experiments show good results in the domain of marine surveillance sequences.
---
paper_title: Evaluation of Maritime Vision Techniques for Aerial Search of Humans in Maritime Environments

paper_content:

Searching for humans lost in vast stretches of ocean has always been a difficult task. In this paper, a range of machine vision approaches are investigated as candidate tools to mitigate the risk of human fatigue and complacency after long hours performing these kind of search tasks. Our two-phased approach utilises point target detection followed by temporal tracking of these targets. Four different point target detection techniques and two tracking techniques are evaluated. We also evaluate the use of different colour spaces for target detection. This paper has a particular focus on Hidden Markov Model based ::: tracking techniques, which seem best able to incorporate a priori knowledge about the maritime search problem, to improve detection performance.
---
paper_title: Machine vision for detection of the rescue target in the marine casualty

paper_content:

In marine rescue, the detection of the target such as life rafts depends on the visual search by man as yet. However, human eyes are sometimes inadequate owing to a long flight and wide views. In order to carry out the prompt rescue of human life, development of the searching system in place of the human eyes is surely required. This paper deals with a new search method for detection of the rescue target using image processing techniques. To detect the small target in the wide views over the sea, we have proposed a new method including the image processing techniques based on the color information and the composite image sensor which increases about the measurement accuracy and the image processing speed at actual field. At the first step of the study, we attempt to extract the image data of the rescue target with the orange color in an experimental sea. >
---
paper_title: Max-mean and max-median filters for detection of small targets

paper_content:

This paper deals with the problem of detection and tracking of low observable small-targets from a sequence of IR images against structural background and non-stationary clutter. There are many algorithms reported in the open literature for detection and tracking of targets of significant size in the image plane with good results. However, the difficulties of detecting small-targets arise from the fact that they are not easily discernable from clutter. The focus of research in this area is to reduce the false alarm rate to an acceptable level. Triple Temporal Filter reported by Jerry Silverman et. al., is one of the promising algorithms in this are. In this paper, we investigate the usefulness of Max-Mean and Max-Median filters in preserving the edges of clouds and structural backgrounds, which helps in detecting small-targets. Subsequently, anti-mean and anti-median operations result in good performance of detecting targets against moving clutter. The raw image is first filtered by max-mean/max-median filter. Then the filtered output is subtracted from the original image to enhance the potential targets. A thresholding step is incorporated in order to limit the number of potential target pixels. The threshold is obtained by using the statistics of the image. Finally, the thresholded images are accumulated so that the moving target forms a continuous trajectory and can be detected by using the post-processing algorithm. It is assumed that most of the targets occupy a couple of pixels. Head-on moving and maneuvering targets are not considered. These filters have ben tested successfully with the available database and the result are presented.
---
paper_title: Detection filters and algorithm fusion for ATR

paper_content:

Detection involves locating all candidate regions of interest (objects) in a scene independent of the object class with object distortions and contrast differences, etc., present. It is one of the most formidable problems in automatic target recognition, since it involves analysis of every local scene region. We consider new detection algorithms and the fusion of their outputs to reduce the probability of false alarm P/sub FA/ while maintaining high probability of detection P/sub D/. Emphasis is given to detecting obscured targets in infrared imagery.
---
paper_title: Identification and tracking of maritime objects in near-infrared image sequences for collision avoidance

paper_content:

This paper describes the continuing development of an image processing system for use on high-speed passenger ferries. The system automatically identifies objects in a maritime scene and uses the detected motion to alert a human observer to potential collision situations. Three integrated image-processing algorithms, namely an image preprocessor, a motion cue generator, and a target tracker, perform the identification and tracking of maritime objects. The pre-processing filters the image and applies a histogram technique to segment the sea from potential objects of interest. The segmented image is passed to the motion cue generator, which provides motion cues based on the differences between consecutive frames of segmented image data. The target tracker applies dynamic constraints on object motion to solve the correspondence problem, thus increasing the confidence that an identified object is a target. Identified and tracked objects are highlighted to a human observer using a white box viewing cue placed directly around the object of interest.
---
paper_title: Nautical Scene Segmentation Using Variable Size Image Windows and Feature Space Reclustering

paper_content:

This paper describes the development of a system for the segmentation of small vessels and objects present in a maritime environment. The system assumes no a priori knowledge of the sea, but uses statistical analysis within variable size image windows to determine a characteristic vector that represents the current sea state. A space of characteristic vectors is searched and a main group of characteristic vectors and its centroid found automatically by using a new method of iterative reclustering. This method is an extension and improvement of the work described in [9]. A Mahalanobis distance measure from the centroid is calculated for each characteristic vector and is used to determine inhomogenities in the sea caused by the presence of a rigid object. The system has been tested using several input image sequences of static small objects such as buoys and small and large maritime vessels moving into and out of a harbour scene and the system successfully segmented these objects.
---
paper_title: Image quality assessment: from error visibility to structural similarity

paper_content:

Objective methods for assessing perceptual image quality traditionally attempted to quantify the visibility of errors (differences) between a distorted image and a reference image using a variety of known properties of the human visual system. Under the assumption that human visual perception is highly adapted for extracting structural information from a scene, we introduce an alternative complementary framework for quality assessment based on the degradation of structural information. As a specific example of this concept, we develop a structural similarity index and demonstrate its promise through a set of intuitive examples, as well as comparison to both subjective ratings and state-of-the-art objective methods on a database of images compressed with JPEG and JPEG2000. A MATLAB implementation of the proposed algorithm is available online at http://www.cns.nyu.edu//spl sim/lcv/ssim/.
---
paper_title: Discriminating small extended targets at sea from clutter and other classes of boats in infrared and visual light imagery

paper_content:

Operating in a coastal environment, with a multitude of boats of different sizes, detection of small extended targets is only one problem. A further difficulty is in discriminating detections of possible threats from alarms due to sea and coastal clutter, and from boats that are neutral for a specific operational task. Adding target features to detections allows filtering out clutter before tracking. Features can also be used to add labels resulting from a classification step. Both will help tracking by facilitating association. Labeling and information from features can be an aid to an operator, or can reduce the number of false alarms for more automatic systems. In this paper we present work on clutter reduction and classification of small extended targets from infrared and visual light imagery. Several methods for discriminating between classes of objects were examined, with an emphasis on less complex techniques, such as rules and decision trees. Similar techniques can be used to discriminate between targets and clutter, and between different classes of boats. Different features are examined that possibly allow discrimination between several classes. Data recordings are used, in infrared and visual light, with a range of targets including rhibs, cabin boats and jet-skis.
---
paper_title: Algorithms for Visual Maritime Surveillance with Rapidly Moving Camera

paper_content:

Visual surveillance in the maritime domain has been explored for more than a decade. Although it has produced a number of working systems and resulted in a mature technology, surveillance has been restricted to the port facilities or areas close to the coastline assuming a fixed-camera scenario. This dissertation presents several contributions in the domain of maritime surveillance. First, a novel algorithm for open-sea visual maritime surveillance is introduced. We explore a challenging situation with a camera mounted on a buoy or other floating platform. The developed algorithm detects, localizes, and tracks ships in the field of view of the camera. Specifically, our method is uniquely designed to handle a rapidly moving camera. Its performance is robust in the presence of a random relatively-large camera motion. In the context of ship detection, a new horizon detection scheme for a complex maritime domain is also developed. Second, the performance of the ship detection algorithm is evaluated on a dataset of 55,000 images. Accuracy of detection of up to 88% of ships is achieved. Lastly, we consider the topic of detection of the vanishing line of the ocean surface plane as a way to estimate the horizon in difficult situations. This allows extension of the ship-detection algorithm to beyond open-sea scenarios.
---
paper_title: Image quality assessment: from error visibility to structural similarity

paper_content:

Objective methods for assessing perceptual image quality traditionally attempted to quantify the visibility of errors (differences) between a distorted image and a reference image using a variety of known properties of the human visual system. Under the assumption that human visual perception is highly adapted for extracting structural information from a scene, we introduce an alternative complementary framework for quality assessment based on the degradation of structural information. As a specific example of this concept, we develop a structural similarity index and demonstrate its promise through a set of intuitive examples, as well as comparison to both subjective ratings and state-of-the-art objective methods on a database of images compressed with JPEG and JPEG2000. A MATLAB implementation of the proposed algorithm is available online at http://www.cns.nyu.edu//spl sim/lcv/ssim/.
---
paper_title: Tracking Ships from Fast Moving Camera through Image Registration

paper_content:

This paper presents an algorithm that detects and tracks marine vessels in video taken by a nonstationary camera installed on an untethered buoy. The video is characterized by large inter-frame motion of the camera, cluttered background, and presence of compression artifacts. Our approach performs segmentation of ships in individual frames processed with a color-gradient filter. The threshold selection is based on the histogram of the search region. Tracking of ships in a sequence is enabled by registering the horizon images in one coordinate system and by using a multihypothesis framework. Registration step uses an area-based technique to correlate a processed strip of the image over the found horizon line. The results of evaluation of detection, localization, and tracking of the ships show significant increase in performance in comparison to the previously used technique.
---
paper_title: Small infrared target fusion detection based on support vector machines in the wavelet domain

paper_content:

A novel method for fusion detection of small infrared targets based on support vector machines (SVM) in the wavelet domain is presented. Target detection task plays an important role in automatic target recognition (ATR) systems because overall ATR performance depends closely on detection results. SVM is a powerful methodology for solving problems in nonlinear classification, function estimation and density estimation. Least-squares support vector machines (LS-SVMs) are reformulations to standard SVMs. The proposed algorithm can be divided into four steps. First, each frame of the image sequence is decomposed by the discrete wavelet frame (DWF). Second, the components with low frequency are performed by regression based on LS-SVM. The one-order partial derivatives in row and column directions are derived. Therefore, feature images of the gradient strength can be obtained. Third, feature images of five consecutive frames are fused to accumulate the energy of target of interest and greatly reduce false alarms. Finally, the segmentation method based on contrast between target and background is utilized to extract the target. In terms of connectivity of moving targets, the majority of residual clutter and false alarms that survive are removed based on 3-D morphological dilation across three consecutive frames along the motion direction of the moving targets. Actual infrared image sequences in backgrounds of sea and sky are applied to validate the proposed approach. Experimental results demonstrate the robustness of the proposed method with high performance.
---
paper_title: Infrared Image Segmentation by Combining Fractal Geometry with Wavelet Transformation

paper_content:

An infrared image is decomposed into three levels by discrete stationary wavelet transform (DSWT). Noise is reduced by wiener filter in the high resolution levels in the DSWT domain. Nonlinear gray transformation operation is used to enhance details in the low resolution levels in the DSWT domain. Enhanced infrared image is obtained by inverse DSWT. The enhanced infrared image is divided into many small blocks. The fractal dimensions of all the blocks are computed. Region of interest (ROI) is extracted by combining all the blocks, which have similar fractal dimensions. ROI is segmented by global threshold method. The man-made objects are efficiently separated from the infrared image by the proposed method. Copyright © 2014 IFSA Publishing, S. L.
---
paper_title: Aquatic debris monitoring using smartphone-based robotic sensors

paper_content:

Monitoring aquatic debris is of great interest to the ecosystems, marine life, human health, and water transport. This paper presents the design and implementation of SOAR - a vision-based surveillance robot system that integrates an off-the-shelf Android smartphone and a gliding robotic fish for debris monitoring. SOAR features real-time debris detection and coverage-based rotation scheduling algorithms. The image processing algorithms for debris detection are specifically designed to address the unique challenges in aquatic environments. The rotation scheduling algorithm provides effective coverage of sporadic debris arrivals despite camera's limited angular view. Moreover, SOAR is able to dynamically offload computation-intensive processing tasks to the cloud for battery power conservation. We have implemented a SOAR prototype and conducted extensive experimental evaluation. The results show that SOAR can accurately detect debris in the presence of various environment and system dynamics, and the rotation scheduling algorithm enables SOAR to capture debris arrivals with reduced energy consumption.
---
paper_title: Segmentation of FLIR images by target enhancement and image model

paper_content:

A new segmentation algorithm of forward-looking infrared (FLIR) images is presented, which first uses median subtraction filter to enhance targets and suppress backgrounds, then uses MBS algorithm to perform segmentation. This algorithm can obtain a precise and accurate segmentation of a target from low contrast FLIR images in complex background. Experimental results compared with MBS algorithm are given. We find that the proposed algorithm shows much better segmentation performance than MBS algorithm in complex background.© (1998) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.
---
paper_title: Robust real-time ship detection and tracking for visual surveillance of cage aquaculture

paper_content:

Abstract This paper presents a visual surveillance scheme for cage aquaculture that automatically detects and tracks ships (intruders). For ship detection and tracking, we propose a robust foreground detection and background updating to effectively reduce the influence of sea waves. Furthermore, we propose a fast 4-connected component labeling method to greatly reduce the computational cost associated with the conventional method. Wave ripples are removed from regions with ships. An improved full search algorithm based on adaptive template block matching with a wave ripple removal is presented to quickly, accurately, and reliably track overlapping ships whose scales change. Experimental results demonstrate that the proposed schemes have outstanding performance in ship detection and tracking. The proposed visual surveillance system for cage aquaculture triggers an alarm if intruders are detected. The security of cage aquaculture can be increased. The proposed visual surveillance can thus greatly help the popularization of cage aquaculture for ocean farming.
---
paper_title: Detecting Moving Objects, Ghosts, and Shadows in Video Streams

paper_content:

Background subtraction methods are widely exploited for moving object detection in videos in many applications, such as traffic monitoring, human motion capture, and video surveillance. How to correctly and efficiently model and update the background model and how to deal with shadows are two of the most distinguishing and challenging aspects of such approaches. The article proposes a general-purpose method that combines statistical assumptions with the object-level knowledge of moving objects, apparent objects (ghosts), and shadows acquired in the processing of the previous frames. Pixels belonging to moving objects, ghosts, and shadows are processed differently in order to supply an object-based selective update. The proposed approach exploits color information for both background subtraction and shadow detection to improve object segmentation and background update. The approach proves fast, flexible, and precise in terms of both pixel accuracy and reactivity to background changes.
---
paper_title: Detection and tracking of moving objects in a maritime environment using level set with shape priors

paper_content:

Over the years, maritime surveillance has become increasingly important due to the recurrence of piracy. While surveillance has traditionally been a manual task using crew members in lookout positions on parts of the ship, much work is being done to automate this task using digital cameras coupled with a computer that uses image processing techniques that intelligently track object in the maritime environment. One such technique is level set segmentation which evolves a contour to objects of interest in a given image. This method works well but gives incorrect segmentation results when a target object is corrupted in the image. This paper explores the possibility of factoring in prior knowledge of a ship’s shape into level set segmentation to improve results, a concept that is unaddressed in maritime surveillance problem. It is shown that the developed video tracking system outperforms level set-based systems that do not use prior shape knowledge, working well even where these systems fail.
---
paper_title: Region-based Mixture of Gaussians modelling for foreground detection in dynamic scenes

paper_content:

One of the most widely used techniques in computer vision for foreground detection is to model each background pixel as a Mixture of Gaussians (MoG). While this is effective for a static camera with a fixed or a slowly varying background, it fails to handle any fast, dynamic movement in the background. In this paper, we propose a generalised framework, called region-based MoG (RMoG), that takes into consideration neighbouring pixels while generating the model of the observed scene. The model equations are derived from expectation maximisation theory for batch mode, and stochastic approximation is used for online mode updates. We evaluate our region-based approach against ten sequences containing dynamic backgrounds, and show that the region-based approach provides a performance improvement over the traditional single pixel MoG. For feature and region sizes that are equal, the effect of increasing the learning rate is to reduce both true and false positives. Comparison with four state-of-the art approaches shows that RMoG outperforms the others in reducing false positives whilst still maintaining reasonable foreground definition. Lastly, using the ChangeDetection (CDNet 2014) benchmark, we evaluated RMoG against numerous surveillance scenes and found it to be amongst the leading performers for dynamic background scenes, whilst providing comparable performance for other commonly occurring surveillance scenes. HighlightsA generalised framework for region based mixture modelling is proposed.Batch and online update equations are derived using expectation maximisation theory.Region-based Mixture of Gaussians (RMoG) algorithm for dynamic background subtractionExperiments show state-of-the-art results in subtracting dynamic backgrounds.
---
paper_title: Argos - a Video Surveillance System for boat Traffic Monitoring in Venice

paper_content:

Visual surveillance in dynamic scenes is currently one of the most active research topics in computer vision, many existing applications are available. However, difficulties in realizing effective video surveillance systems that are robust to the many different conditions that arise in real environments, make the actual deployment of such systems very challenging. In this article, we present a real, unique and pioneer video surveillance system for boat traffic monitoring, ARGOS. The system runs continuously 24 hours a day, 7 days a week, day and night in the city of Venice (Italy) since 2007 and it is able to build a reliable background model of the water channel and to track the boats navigating the channel with good accuracy in real-time. A significant experimental evaluation, reported in this article, has been performed in order to assess the real performance of the system.
---
paper_title: A practical adaptive approach for dynamic background subtraction using an invariant colour model and object tracking

paper_content:

In this paper, three dynamic background subtraction algorithms for colour images are presented and compared. The performances of these algorithms defined as 'Selective Update using Temporal Averaging', 'Selective Update using Non-foreground Pixels of the Input Image' and 'Selective Update using Temporal Median' are only different for background pixels. Then using an invariant colour filter and a suitable motion tracking technique, an object-level classification is offered that recognises the behaviours of all foreground blobs. This novel approach, which selectively excludes foreground blobs from the background frames, is included in all three methods. It is shown that the 'Selective Update using Temporal Median' produces the correct background image for each input frame. The advantages of the third algorithm are: it operates in unconstrained outdoor and indoor scenes. Also it is able to handle difficult situations such as removing ghosts and including stationary objects in the background image efficiently. Meanwhile, the algorithm's parameters are computed automatically or are fixed. The efficiency of the new algorithm is confirmed by the results obtained on a number of image sequences.
---
paper_title: Wallflower: principles and practice of background maintenance

paper_content:

Background maintenance is a frequent element of video surveillance systems. We develop Wallflower, a three-component system for background maintenance: the pixel-level component performs Wiener filtering to make probabilistic predictions of the expected background; the region-level component fills in homogeneous regions of foreground objects; and the frame-level component detects sudden, global changes in the image and swaps in better approximations of the background. We compare our system with 8 other background subtraction algorithms. Wallflower is shown to outperform previous algorithms by handling a greater set of the difficult situations that can occur. Finally, we analyze the experimental results and propose normative principles for background maintenance.
---
paper_title: Towards robust automatic traffic scene analysis in real-time

paper_content:

Automatic symbolic traffic scene analysis is essential to many areas of IVHS (Intelligent Vehicle Highway Systems). Traffic scene information can be used to optimize traffic flow during busy periods, identify stalled vehicles and accidents, and aid the decision-making of an autonomous vehicle controller. Improvements in technologies for machine vision-based surveillance and high-level symbolic reasoning have enabled the authors to develop a system for detailed, reliable traffic scene analysis. The machine vision component of the system employs a contour tracker and an affine motion model based on Kalman filters to extract vehicle trajectories over a sequence of traffic scene images. The symbolic reasoning component uses a dynamic belief network to make inferences about traffic events such as vehicle lane changes and stalls. In this paper, the authors discuss the key tasks of the vision and reasoning components as well as their integration into a working prototype. Preliminary results of an implementation on special purpose hardware using C-40 Digital Signal Processors show that near real-time performance can be achieved without further improvements.
---
paper_title: Outdoor infrared video surveillance: A novel dynamic technique for the subtraction of a changing background of IR images

paper_content:

Abstract For security applications, automatic detection and tracking of moving objects is an important and challenging issue especially in uncontrolled environments. Recently, due to the decreasing costs and increasing miniaturization of infrared sensors, the use of infrared imaging technology has become an interesting alternative in such applications. In this paper, a framework is proposed to detect, track and classify both pedestrians and vehicles in realistic scenarios using a stationary infrared camera. More specifically, a novel dynamic background-subtraction technique to robustly adapt detection to illumination changes in outdoor scenes is proposed. We noticed that combining results with edge detection enables to reduce considerably false alarms while this reinforces also tracking efficiency. The proposed system was implemented and tested successfully in various environmental conditions.
---
paper_title: Polynomial background estimation using visible light video streams for robust automatic detection in a maritime environment

paper_content:

For naval surveillance, automatic detection of surface objects, like vessels, in a maritime environment is an ::: important contribution of the Electro-Optical (EO) sensor systems on board. Based on previous research using ::: single images, a background estimation approach using low-order polynomials is proposed for the automatic ::: detection of objects in a maritime environment. The polynomials are fitted to the intensity values in the image ::: after which the deviation between the fitted intensity values and the measured intensity values are used for detection. The research presented in this paper, includes the time information by using video streams instead of single images. Hereby, the level of fusing time information and the number of frames necessary for stable detection and tracking behaviour are analysed and discussed. The performance of the detection approach is tested on a, during the fall of 2007, collected extensive dataset of maritime pictures in the Mediterranean Sea and in the North Sea on board of an Air Defence Command frigate, HNLMS Tromp.
---
paper_title: Identification and tracking of maritime objects in near-infrared image sequences for collision avoidance

paper_content:

This paper describes the continuing development of an image processing system for use on high-speed passenger ferries. The system automatically identifies objects in a maritime scene and uses the detected motion to alert a human observer to potential collision situations. Three integrated image-processing algorithms, namely an image preprocessor, a motion cue generator, and a target tracker, perform the identification and tracking of maritime objects. The pre-processing filters the image and applies a histogram technique to segment the sea from potential objects of interest. The segmented image is passed to the motion cue generator, which provides motion cues based on the differences between consecutive frames of segmented image data. The target tracker applies dynamic constraints on object motion to solve the correspondence problem, thus increasing the confidence that an identified object is a target. Identified and tracked objects are highlighted to a human observer using a white box viewing cue placed directly around the object of interest.
---
paper_title: Detection and tracking of ships in open sea with rapidly moving buoy-mounted camera system

paper_content:

Abstract Visual surveillance in the maritime domain has been explored for more than a decade. Although it has produced a number of working systems and resulted in a mature technology, surveillance has been restricted to the port facilities or areas close to the coast line assuming a fixed-camera scenario. This paper presents a novel algorithm for open-sea visual maritime surveillance. We explore a challenging situation with a forward-looking camera mounted on a buoy or other floating platform. The proposed algorithm detects, localizes, and tracks ships in the field of view of the camera. Specifically, developed algorithm is uniquely designed to handle rapidly moving camera. Its performance is robust in the presence of a random relatively-large camera motion. In the context of ship detection we developed a new horizon detection scheme for a complex maritime domain. The performance of our algorithm and its comprising elements is evaluated. Ship detection precision of 88% is achieved on a large dataset collected from a prototype system.
---
paper_title: Adaptive maritime video surveillance

paper_content:

Maritime assets such as ports, harbors, and vessels are vulnerable to a variety of near-shore threats such as small-boat attacks. Currently, such vulnerabilities are addressed predominantly by watchstanders and manual video surveillance, which is manpower intensive. Automatic maritime video surveillance techniques are being introduced to reduce manpower costs, but they have limited functionality and performance. For example, they only detect simple events such as perimeter breaches and cannot predict emerging threats. They also generate too many false alerts and cannot explain their reasoning. To overcome these limitations, we are developing the Maritime Activity Analysis Workbench (MAAW), which will be a mixed-initiative real-time maritime video surveillance tool that uses an integrated supervised machine learning approach to label independent and coordinated maritime activities. It uses the same information to predict anomalous behavior and explain its reasoning; this is an important capability for watchstander training and for collecting performance feedback. In this paper, we describe MAAW's functional architecture, which includes the following pipeline of components: (1) a video acquisition and preprocessing component that detects and tracks vessels in video images, (2) a vessel categorization and activity labeling component that uses standard and relational supervised machine learning methods to label maritime activities, and (3) an ontology-guided vessel and maritime activity annotator to enable subject matter experts (e.g., watchstanders) to provide feedback and supervision to the system. We report our findings from a preliminary system evaluation on river traffic video.
---
paper_title: Detection of Dynamic Background Due to Swaying Movements From Motion Features

paper_content:

Dynamically changing background (dynamic background) still presents a great challenge to many motion-based video surveillance systems. In the context of event detection, it is a major source of false alarms. There is a strong need from the security industry either to detect and suppress these false alarms, or dampen the effects of background changes, so as to increase the sensitivity to meaningful events of interest. In this paper, we restrict our focus to one of the most common causes of dynamic background changes: 1) that of swaying tree branches and 2) their shadows under windy conditions. Considering the ultimate goal in a video analytics pipeline, we formulate a new dynamic background detection problem as a signal processing alternative to the previously described but unreliable computer vision-based approaches. Within this new framework, we directly reduce the number of false alarms by testing if the detected events are due to characteristic background motions. In addition, we introduce a new data set suitable for the evaluation of dynamic background detection. It consists of real-world events detected by a commercial surveillance system from two static surveillance cameras. The research question we address is whether dynamic background can be detected reliably and efficiently using simple motion features and in the presence of similar but meaningful events, such as loitering. Inspired by the tree aerodynamics theory, we propose a novel method named local variation persistence (LVP), that captures the key characteristics of swaying motions. The method is posed as a convex optimization problem, whose variable is the local variation. We derive a computationally efficient algorithm for solving the optimization problem, the solution of which is then used to form a powerful detection statistic. On our newly collected data set, we demonstrate that the proposed LVP achieves excellent detection results and outperforms the best alternative adapted from existing art in the dynamic background literature.
---
paper_title: Wavelet transform methods for object detection and recovery

paper_content:

We show that a biorthogonal spline wavelet closely approximates the prewhitening matched filter for detecting Gaussian objects in Markov noise. The filterbank implementation of the wavelet transform acts as a hierarchy of such detectors operating at discrete object scales. If the object to be detected is Gaussian and its scale happens to coincide with one of those computed by the wavelet transform, and if the background noise is truly Markov, then optimum detection is realized by thresholding the appropriate subband image. In reality, the Gaussian may be a rather coarse approximation of the object, and the background noise may deviate from the Markov assumption. In this case, we may view the wavelet decomposition as a means for computing an orthogonal feature set for input to a classifier. We use a supervised linear classifier applied to feature vectors comprised of samples taken from the subbands of an N-octave, undecimated wavelet transform. The resulting map of test statistic values indicates the presence and location of objects. The object itself is reconstructed by using the test statistic to emphasize wavelet subbands, followed by computing the inverse wavelet transform. We show two contrasting applications of the wavelets-based object recovery algorithm. For detecting microcalcifications in digitized mammograms, the object and noise models closely match the real image data, and the multiscale matched filter paradigm is highly appropriate. The second application, extracting ship outlines in noisy forward-looking infrared images, is presented as a case where good results are achieved despite the data models being less well matched to the assumptions of the algorithm.
---
paper_title: Video object extraction based on adaptive background and statistical change detection

paper_content:

This paper introduces a system for video object extraction useful for general applications where foreground objects move within a slow changing background. Surveillance of indoor and outdoor sequences is a typical example. The originality of the approach resides in two related components. First, the statistical change detection used in the system does not require any sophisticated parametric tuning as it is based on a probabilistic method. Second, the change is detected between a current instance of the scene and a reference that is updated continuously to take into account slow variation of the background. Simulation results show that the proposed scheme performs well in extracting video objects, with stability and good accuracy, while being of relative reduced complexity.
---
paper_title: Target identification in a complex maritime scene

paper_content:

This paper describes new methods applied to the problem of target identification in complex maritime scenes. The algorithm described here automatically identifies rigid objects moving in a maritime scene. The methods for target identification proposed here are based on the change in statistical descriptors of image segments and on axis projections of the difference image. Results from this stage of processing are passed to a decision maker which determines the resulting regions of interest in the image. Different sequences of maritime scenes were passed to the algorithm and results show that it was capable of identifying most of the rigid objects moving in the scenes despite the variation of the scenes.
---
paper_title: Motion-based background subtraction using adaptive kernel density estimation

paper_content:

Background modeling is an important component of many vision systems. Existing work in the area has mostly addressed scenes that consist of static or quasi-static structures. When the scene exhibits a persistent dynamic behavior in time, such an assumption is violated and detection performance deteriorates. In this paper, we propose a new method for the modeling and subtraction of such scenes. Towards the modeling of the dynamic characteristics, optical flow is computed and utilized as a feature in a higher dimensional space. Inherent ambiguities in the computation of features are addressed by using a data-dependent bandwidth for density estimation using kernels. Extensive experiments demonstrate the utility and performance of the proposed approach.
---
paper_title: Background models for tracking objects in water

paper_content:

This paper presents a novel background analysis technique to enable robust tracking of objects in water- based scenarios. Current pixel-wise statistical background models support automatic change detection in many outdoor situations, but are limited to background changes which can be modeled via a set of per-pixel spatially uncorrelated processes. In water-based scenarios, waves caused by wind or by moving vessels (wakes) form highly correlated moving patterns that confuse traditional background analysis models. In this work we introduce a framework that explicitly models this type of background variation. The framework combines the output of a statistical background model with localized optical flow analysis to produce two motion maps. In the final stage we apply object-level fusion to filter out moving regions that are most likely caused by wave clutter. A tracking algorithm can now handle the resulting set of objects.
---
paper_title: Background modeling in the maritime domain

paper_content:

Maritime environment represents a challenging scenario for automatic video surveillance due to the complexity of the observed scene: waves on the water surface, boat wakes, and weather issues contribute to generate a highly dynamic background. Moreover, an appropriate background model has to deal with gradual and sudden illumination changes, camera jitter, shadows, and reflections that can provoke false detections. Using a predefined distribution (e.g., Gaussian) for generating the background model can result ineffective, due to the need of modeling non-regular patterns. In this paper, a method for creating a "discretization" of an unknown distribution that can model highly dynamic background such as water is described. A quantitative evaluation carried out on two publicly available datasets of videos and images, containing data recorded in different maritime scenarios, with varying light and weather conditions, demonstrates the effectiveness of the approach.
---
paper_title: A Hybrid Color-Based Foreground Object Detection Method for Automated Marine Surveillance

paper_content:

This paper proposes a hybrid foreground object detection method suitable for the marine surveillance applications. Our approach combines an existing foreground object detection method with an image color segmentation technique to improve accuracy. The foreground segmentation method employs a Bayesian decision framework, while the color segmentation part is graph-based and relies on the local variation of edges. We also establish the set of requirements any practical marine surveillance algorithm should fulfill, and show that our method conforms to these requirements. Experiments show good results in the domain of marine surveillance sequences.
---
paper_title: Detecting Moving Objects, Ghosts, and Shadows in Video Streams

paper_content:

Background subtraction methods are widely exploited for moving object detection in videos in many applications, such as traffic monitoring, human motion capture, and video surveillance. How to correctly and efficiently model and update the background model and how to deal with shadows are two of the most distinguishing and challenging aspects of such approaches. The article proposes a general-purpose method that combines statistical assumptions with the object-level knowledge of moving objects, apparent objects (ghosts), and shadows acquired in the processing of the previous frames. Pixels belonging to moving objects, ghosts, and shadows are processed differently in order to supply an object-based selective update. The proposed approach exploits color information for both background subtraction and shadow detection to improve object segmentation and background update. The approach proves fast, flexible, and precise in terms of both pixel accuracy and reactivity to background changes.
---
paper_title: Foreground object detection from videos containing complex background

paper_content:

This paper proposes a novel method for detection and segmentation of foreground objects from a video which contains both stationary and moving background objects and undergoes both gradual and sudden "once-off" changes. A Bayes decision rule for classification of background and foreground from selected feature vectors is formulated. Under this rule, different types of background objects will be classified from foreground objects by choosing a proper feature vector. The stationary background object is described by the color feature, and the moving background object is represented by the color co-occurrence feature. Foreground objects are extracted by fusing the classification results from both stationary and moving pixels. Learning strategies for the gradual and sudden "once-off" background changes are proposed to adapt to various changes in background through the video. The convergence of the learning process is proved and a formula to select a proper learning rate is also derived. Experiments have shown promising results in extracting foreground objects from many complex backgrounds including wavering tree branches, flickering screens and water surfaces, moving escalators, opening and closing doors, switching lights and shadows of moving objects.
---
paper_title: Daytime Water Detection by Fusing Multiple Cues for Autonomous Off-Road Navigation

paper_content:

Abstract : Detecting water hazards is a significant challenge to unmanned ground vehicle autonomous off-road navigation. This paper focuses on detecting the presence of water during the daytime using color cameras. A multi-cue approach is taken. Evidence of the presence of water is generated from color, texture, and the detection of reflections in stereo range data. A rule base for fusing water cues was developed by evaluating detection results from an extensive archive of data collection imagery containing water. This software has been implemented into a run-time passive perception subsystem and tested thus far under Linux on a Pentium based processor.
---
paper_title: Background modeling in the maritime domain

paper_content:

Maritime environment represents a challenging scenario for automatic video surveillance due to the complexity of the observed scene: waves on the water surface, boat wakes, and weather issues contribute to generate a highly dynamic background. Moreover, an appropriate background model has to deal with gradual and sudden illumination changes, camera jitter, shadows, and reflections that can provoke false detections. Using a predefined distribution (e.g., Gaussian) for generating the background model can result ineffective, due to the need of modeling non-regular patterns. In this paper, a method for creating a "discretization" of an unknown distribution that can model highly dynamic background such as water is described. A quantitative evaluation carried out on two publicly available datasets of videos and images, containing data recorded in different maritime scenarios, with varying light and weather conditions, demonstrates the effectiveness of the approach.
---
paper_title: Segmentation and tracking of piglets in images

paper_content:

An algorithm was developed for the segmentation and tracking of piglets and tested on a 200-image sequence of 10 piglets moving on a straw background. The image-capture rate was 1 image/140 ms. The segmentation method was a combination of image differencing with respect to a median background and a Laplacian operator. The features tracked were blob edges in the segmented image. During tracking, the piglets were modelled as ellipses initialised on the blobs. Each piglet was tracked by searching for blob edges in an elliptical window about the piglet's position, which was predicted from its previous two positions.
---
paper_title: Efficient adaptive density estimation per image pixel for the task of background subtraction

paper_content:

We analyze the computer vision task of pixel-level background subtraction. We present recursive equations that are used to constantly update the parameters of a Gaussian mixture model and to simultaneously select the appropriate number of components for each pixel. We also present a simple non-parametric adaptive density estimation method. The two methods are compared with each other and with some previously proposed algorithms.
---
paper_title: An iterative image registration technique with an application to stereo vision

paper_content:

Image registration finds a variety of applications in computer vision. Unfortunately, traditional image registration techniques tend to be costly. We present a new image registration technique that makes use of the spatial intensity gradient of the images to find a good match using a type of Newton-Raphson iteration. Our technique is taster because it examines far fewer potential matches between the images than existing techniques Furthermore, this registration technique can be generalized to handle rotation, scaling and shearing. We show how our technique can be adapted tor use in a stereo vision system.
---
paper_title: Non-parametric model for background subtraction

paper_content:

Background subtraction is a method typically used to segment moving regions in image sequences taken from a static camera by comparing each new frame to a model of the scene background. We present a novel non-parametric background model and a background subtraction approach. The model can handle situations where the background of the scene is cluttered and not completely static but contains small motions such as tree branches and bushes. The model estimates the probability of observing pixel intensity values based on a sample of intensity values for each pixel. The model adapts quickly to changes in the scene which enables very sensitive detection of moving targets. We also show how the model can use color information to suppress detection of shadows. The implementation of the model runs in real-time for both gray level and color imagery. Evaluation shows that this approach achieves very sensitive detection with very low false alarm rates.
---
paper_title: Robust real-time ship detection and tracking for visual surveillance of cage aquaculture

paper_content:

Abstract This paper presents a visual surveillance scheme for cage aquaculture that automatically detects and tracks ships (intruders). For ship detection and tracking, we propose a robust foreground detection and background updating to effectively reduce the influence of sea waves. Furthermore, we propose a fast 4-connected component labeling method to greatly reduce the computational cost associated with the conventional method. Wave ripples are removed from regions with ships. An improved full search algorithm based on adaptive template block matching with a wave ripple removal is presented to quickly, accurately, and reliably track overlapping ships whose scales change. Experimental results demonstrate that the proposed schemes have outstanding performance in ship detection and tracking. The proposed visual surveillance system for cage aquaculture triggers an alarm if intruders are detected. The security of cage aquaculture can be increased. The proposed visual surveillance can thus greatly help the popularization of cage aquaculture for ocean farming.
---
paper_title: Detection and tracking of moving objects in a maritime environment using level set with shape priors

paper_content:

Over the years, maritime surveillance has become increasingly important due to the recurrence of piracy. While surveillance has traditionally been a manual task using crew members in lookout positions on parts of the ship, much work is being done to automate this task using digital cameras coupled with a computer that uses image processing techniques that intelligently track object in the maritime environment. One such technique is level set segmentation which evolves a contour to objects of interest in a given image. This method works well but gives incorrect segmentation results when a target object is corrupted in the image. This paper explores the possibility of factoring in prior knowledge of a ship’s shape into level set segmentation to improve results, a concept that is unaddressed in maritime surveillance problem. It is shown that the developed video tracking system outperforms level set-based systems that do not use prior shape knowledge, working well even where these systems fail.
---
paper_title: Multispectral Target Detection and Tracking for Seaport Video Surveillance

paper_content:

In this paper, a video surveillance process is presented including target detection and tracking of ships at the entrance of a seaport in order to improve security and to prevent terrorist attacks. This process is helpful in the automatic analysis of movements inside the seaport. Steps of detection and tracking are completed using IR data whereas the pattern recognition stage is achieved on color data. A comparison of results of detection and tracking is presented on both IR and color data in order to justify the choice of IR images for these two steps. A draft description of the pattern recognition stage is finally drawn up as development prospect.
---
paper_title: Evaluation of Maritime Vision Techniques for Aerial Search of Humans in Maritime Environments

paper_content:

Searching for humans lost in vast stretches of ocean has always been a difficult task. In this paper, a range of machine vision approaches are investigated as candidate tools to mitigate the risk of human fatigue and complacency after long hours performing these kind of search tasks. Our two-phased approach utilises point target detection followed by temporal tracking of these targets. Four different point target detection techniques and two tracking techniques are evaluated. We also evaluate the use of different colour spaces for target detection. This paper has a particular focus on Hidden Markov Model based ::: tracking techniques, which seem best able to incorporate a priori knowledge about the maritime search problem, to improve detection performance.
---
paper_title: Machine vision for detection of the rescue target in the marine casualty

paper_content:

In marine rescue, the detection of the target such as life rafts depends on the visual search by man as yet. However, human eyes are sometimes inadequate owing to a long flight and wide views. In order to carry out the prompt rescue of human life, development of the searching system in place of the human eyes is surely required. This paper deals with a new search method for detection of the rescue target using image processing techniques. To detect the small target in the wide views over the sea, we have proposed a new method including the image processing techniques based on the color information and the composite image sensor which increases about the measurement accuracy and the image processing speed at actual field. At the first step of the study, we attempt to extract the image data of the rescue target with the orange color in an experimental sea. >
---
paper_title: Persistent maritime surveillance using multi-sensor feature association and classification

paper_content:

In maritime operational scenarios, such as smuggling, piracy, or terrorist threats, it is not only relevant who or what an observed object is, but also where it is now and in the past in relation to other (geographical) objects. In situation and impact assessment, this information is used to determine whether an object is a threat. Single platform (ship, harbor) or single sensor information will not provide all this information. The work presented in this paper focuses on the sensor and object levels that provide a description of currently observed objects to situation assessment. For use of information of objects at higher information levels, it is necessary to have not only a good description of observed objects at this moment, but also from its past. Therefore, currently observed objects have to be linked to previous occurrences. Kinematic features, as used in tracking, are of limited use, as uncertainties over longer time intervals are so large that no unique associations can be made. Features extracted from different sensors (e.g., ESM, EO/IR) can be used for both association and classification. Features and classifications are used to associate current objects to previous object descriptions, allowing objects to be described better, and provide position history. In this paper a description of a high level architecture in which such a multi-sensor association is used is described. Results of an assessment of the usability of several features from ESM (from spectrum), EO and IR (shape, contour, keypoints) data for association and classification are shown. © 2012 Copyright Society of Photo-Optical Instrumentation Engineers (SPIE).
---
paper_title: Maritime Surveillance : Tracking Ships inside a Dynamic Background Using a Fast Level-Set

paper_content:

Research highlights? Existing vision-based tracking methods are not suitable for the maritime domain. ? We derive a suitable tracking method by combining and modifying existing methods. ? Our method can track tiny targets. ? Our method is validated on several test sequences and two live field trials. Surveillance in a maritime environment is indispensable in the fight against a wide range of criminal activities, including pirate attacks, unlicensed fishing trailers and human trafficking. Computer vision systems can be a useful aid in the law enforcement process, by for example tracking and identifying moving vessels on the ocean. However, the maritime domain poses many challenges for the design of an effective maritime surveillance system. One such challenge is the tracking of moving vessels in the presence of a moving dynamic background (the ocean). We present techniques that address this particular problem. We use a background subtraction method and employ a real-time approximation of level-set-based curve evolution to demarcate the outline of moving vessels in the ocean. We report promising results on both small and large vessels, based on two field trials.
---
paper_title: AUTOMATIC MARITIME SURVEILLANCE WITH VISUAL TARGET DETECTION

paper_content:

In this paper an automatic maritime surveillance system is presented. Boat detection is performed by means of an Haar-like classifier in order to obtain robustness with respect to targets having very different size, reflections and wakes on the water surface, and apparently motionless boats anchored off the coast. Detection results are filtered over the time in order to reduce the false alarm rate. Experimental results show the effectiveness of the approach with different light conditions and camera positions. The system is able to provide the user a global view adding a visual dimension to AIS data.
---
paper_title: Evaluation of Maritime Vision Techniques for Aerial Search of Humans in Maritime Environments

paper_content:

Searching for humans lost in vast stretches of ocean has always been a difficult task. In this paper, a range of machine vision approaches are investigated as candidate tools to mitigate the risk of human fatigue and complacency after long hours performing these kind of search tasks. Our two-phased approach utilises point target detection followed by temporal tracking of these targets. Four different point target detection techniques and two tracking techniques are evaluated. We also evaluate the use of different colour spaces for target detection. This paper has a particular focus on Hidden Markov Model based ::: tracking techniques, which seem best able to incorporate a priori knowledge about the maritime search problem, to improve detection performance.
---
paper_title: Automated intelligent video surveillance system for ships

paper_content:

To protect naval and commercial ships from attack by terrorists and pirates, it is important to have automatic surveillance ::: systems able to detect, identify, track and alert the crew on small watercrafts that might pursue malicious intentions, ::: while ruling out non-threat entities. Radar systems have limitations on the minimum detectable range and lack high-level ::: classification power. In this paper, we present an innovative Automated Intelligent Video Surveillance System for Ships ::: (AIVS3) as a vision-based solution for ship security. Capitalizing on advanced computer vision algorithms and practical ::: machine learning methodologies, the developed AIVS3 is not only capable of efficiently and robustly detecting, ::: classifying, and tracking various maritime targets, but also able to fuse heterogeneous target information to interpret ::: scene activities, associate targets with levels of threat, and issue the corresponding alerts/recommendations to the man-in- ::: the-loop (MITL). AIVS3 has been tested in various maritime scenarios and shown accurate and effective threat ::: detection performance. By reducing the reliance on human eyes to monitor cluttered scenes, AIVS3 will save the ::: manpower while increasing the accuracy in detection and identification of asymmetric attacks for ship protection.
---
paper_title: Argos - a Video Surveillance System for boat Traffic Monitoring in Venice

paper_content:

Visual surveillance in dynamic scenes is currently one of the most active research topics in computer vision, many existing applications are available. However, difficulties in realizing effective video surveillance systems that are robust to the many different conditions that arise in real environments, make the actual deployment of such systems very challenging. In this article, we present a real, unique and pioneer video surveillance system for boat traffic monitoring, ARGOS. The system runs continuously 24 hours a day, 7 days a week, day and night in the city of Venice (Italy) since 2007 and it is able to build a reliable background model of the water channel and to track the boats navigating the channel with good accuracy in real-time. A significant experimental evaluation, reported in this article, has been performed in order to assess the real performance of the system.
---
paper_title: Robust Multiple Car Tracking with Occlusion Reasoning

paper_content:

In this work we address the problem of occlusion in tracking multiple 3D objects in a known environment and propose a new approach for tracking vehicles in road traffic scenes using an explicit occlusion reasoning step. We employ a contour tracker based on intensity and motion boundaries. The motion of the contour of the vehicles in the image is assumed to be well describable by an affine motion model with a translation and a change in scale. A vehicle contour is represented by closed cubic splines the position and motion of which is estimated along the image sequence. In order to employ linear Kalman Filters we decompose the estimation process into two filters: one for estimating the affine motion parameters and one for estimating the shape of the contours of the vehicles. Occlusion detection is performed by intersecting the depth ordered regions associated to the objects. The intersection part is then excluded in the motion and shape estimation. This procedure also improves the shape estimation in case of adjacent objects since occlusion detection is performed on slightly enlarged regions. In this way we obtain robust motion estimates and trajectories for vehicles even in the case of occlusions, as we show in some experiments with real world traffic scenes.
---
paper_title: Contour Tracking By Stochastic Propagation of Conditional Density

paper_content:

The problem of tracking curves in dense visual clutter is a challenging one. Trackers based on Kalman filters are of limited use; because they are based on Gaussian densities which are unimodal, they cannot represent simultaneous alternative hypotheses. Extensions to the Kalman filter to handle multiple data associations work satisfactorily in the simple case of point targets, but do not extend naturally to continuous curves. A new, stochastic algorithm is proposed here, the Condensation algorithm — Conditional Density Propagation over time. It uses ‘factored sampling’, a method previously applied to interpretation of static images, in which the distribution of possible interpretations is represented by a randomly generated set of representatives. The Condensation algorithm combines factored sampling with learned dynamical models to propagate an entire probability distribution for object position and shape, over time. The result is highly robust tracking of agile motion in clutter, markedly superior to what has previously been attainable from Kalman filtering. Notwithstanding the use of stochastic methods, the algorithm runs in near real-time.
---
paper_title: Extended Object Tracking Using Monte Carlo Methods

paper_content:

This correspondence addresses the problem of tracking extended objects, such as ships or a convoy of vehicles moving in urban environment. Two Monte Carlo techniques for extended object tracking are proposed: an interacting multiple model data augmentation (IMM-DA) algorithm and a modified version of the mixture Kalman filter (MKF) of Chen and Liu , called the mixture Kalman filter modified (MKFm). The data augmentation (DA) technique with finite mixtures estimates the object extent parameters, whereas an interacting multiple model (IMM) filter estimates the kinematic states (position and speed) of the manoeuvring object. Next, the system model is formulated in a partially conditional dynamic linear (PCDL) form. This affords us to propose two latent indicator variables characterizing, respectively, the motion mode and object size. Then, an MKFm is developed with the PCDL model. The IMM-DA and the MKFm performance is compared with a combined IMM-particle filter (IMM-PF) algorithm with respect to accuracy and computational complexity. The most accurate parameter estimates are obtained by the DA algorithm, followed by the MKFm and PF.
---
paper_title: Detection and tracking of ships in open sea with rapidly moving buoy-mounted camera system

paper_content:

Abstract Visual surveillance in the maritime domain has been explored for more than a decade. Although it has produced a number of working systems and resulted in a mature technology, surveillance has been restricted to the port facilities or areas close to the coast line assuming a fixed-camera scenario. This paper presents a novel algorithm for open-sea visual maritime surveillance. We explore a challenging situation with a forward-looking camera mounted on a buoy or other floating platform. The proposed algorithm detects, localizes, and tracks ships in the field of view of the camera. Specifically, developed algorithm is uniquely designed to handle rapidly moving camera. Its performance is robust in the presence of a random relatively-large camera motion. In the context of ship detection we developed a new horizon detection scheme for a complex maritime domain. The performance of our algorithm and its comprising elements is evaluated. Ship detection precision of 88% is achieved on a large dataset collected from a prototype system.
---
paper_title: Segmenting foreground objects from a dynamic textured background via a robust Kalman filter

paper_content:

The algorithm presented aims to segment the foreground objects in video (e.g., people) given time-varying, textured backgrounds. Examples of time-varying backgrounds include waves on water, clouds moving, trees waving in the wind, automobile traffic, moving crowds, escalators, etc. We have developed a novel foreground-background segmentation algorithm that explicitly accounts for the nonstationary nature and clutter-like appearance of many dynamic textures. The dynamic texture is modeled by an autoregressive moving average model (ARMA). A robust Kalman filter algorithm iteratively estimates the intrinsic appearance of the dynamic texture, as well as the regions of the foreground objects. Preliminary experiments with this method have demonstrated promising results.
---
paper_title: ficient Implementation of Reid's Multiple ypothesis Tracking Algorithm and Its Evaluation for the Purpose of Visual Tracking

paper_content:

An efficient implementation of Reid's multiple hypothesis tracking (MHT) algorithm is presented in which the k-best hypotheses are determined in polynomial time using an algorithm due to Murly (1968). The MHT algorithm is then applied to several motion sequences. The MHT capabilities of track initiation, termination, and continuation are demonstrated together with the latter's capability to provide low level support of temporary occlusion of tracks. Between 50 and 150 corner features are simultaneously tracked in the image plane over a sequence of up to 51 frames. Each corner is tracked using a simple linear Kalman filter and any data association uncertainty is resolved by the MHT. Kalman filter parameter estimation is discussed, and experimental results show that the algorithm is robust to errors in the motion model. An investigation of the performance of the algorithm as a function of look-ahead (tree depth) indicates that high accuracy can be obtained for tree depths as shallow as three. Experimental results suggest that a real-time MHT solution to the motion correspondence problem is possible for certain classes of scenes.
---
paper_title: Tracking Ships from Fast Moving Camera through Image Registration

paper_content:

This paper presents an algorithm that detects and tracks marine vessels in video taken by a nonstationary camera installed on an untethered buoy. The video is characterized by large inter-frame motion of the camera, cluttered background, and presence of compression artifacts. Our approach performs segmentation of ships in individual frames processed with a color-gradient filter. The threshold selection is based on the histogram of the search region. Tracking of ships in a sequence is enabled by registering the horizon images in one coordinate system and by using a multihypothesis framework. Registration step uses an area-based technique to correlate a processed strip of the image over the found horizon line. The results of evaluation of detection, localization, and tracking of the ships show significant increase in performance in comparison to the previously used technique.
---
paper_title: Motion-based background subtraction using adaptive kernel density estimation

paper_content:

Background modeling is an important component of many vision systems. Existing work in the area has mostly addressed scenes that consist of static or quasi-static structures. When the scene exhibits a persistent dynamic behavior in time, such an assumption is violated and detection performance deteriorates. In this paper, we propose a new method for the modeling and subtraction of such scenes. Towards the modeling of the dynamic characteristics, optical flow is computed and utilized as a feature in a higher dimensional space. Inherent ambiguities in the computation of features are addressed by using a data-dependent bandwidth for density estimation using kernels. Extensive experiments demonstrate the utility and performance of the proposed approach.
---
paper_title: Argos - a Video Surveillance System for boat Traffic Monitoring in Venice

paper_content:

Visual surveillance in dynamic scenes is currently one of the most active research topics in computer vision, many existing applications are available. However, difficulties in realizing effective video surveillance systems that are robust to the many different conditions that arise in real environments, make the actual deployment of such systems very challenging. In this article, we present a real, unique and pioneer video surveillance system for boat traffic monitoring, ARGOS. The system runs continuously 24 hours a day, 7 days a week, day and night in the city of Venice (Italy) since 2007 and it is able to build a reliable background model of the water channel and to track the boats navigating the channel with good accuracy in real-time. A significant experimental evaluation, reported in this article, has been performed in order to assess the real performance of the system.
---
paper_title: Acoustic approaches to remote species identification: a review

paper_content:

Noninvasive species identification remains a longterm goal of fishers, researchers, and resource managers who use sound to locate, map, and count aquatic organisms. Since the first biological applications of underwater acoustics, four approaches have been used singly or in combination to survey marine and freshwater environments: passive sonar; prior knowledge and direct sampling; echo statistics from high-frequency measures; and matching models to low-frequency measures. Echo amplitudes or targets measured using any sonar equipment are variable signals. Variability in reflected sound is influenced by physical factors associated with the transmission of sound through a compressible fluid, and by biological factors associated with the location, reflective properties, and behaviour of a target. The current trend in acoustic target identification is to increase the amount of information collected through increases in frequency bandwidth or in the number of acoustic beams. Exclusive use of acoustics to identify aquatic organisms reliably will require a set of statistical metrics that discriminate among a wide range of similar body types at any packing density, and incorporation of these algorithms in routine data processing.
---
paper_title: An iterative image registration technique with an application to stereo vision

paper_content:

Image registration finds a variety of applications in computer vision. Unfortunately, traditional image registration techniques tend to be costly. We present a new image registration technique that makes use of the spatial intensity gradient of the images to find a good match using a type of Newton-Raphson iteration. Our technique is taster because it examines far fewer potential matches between the images than existing techniques Furthermore, this registration technique can be generalized to handle rotation, scaling and shearing. We show how our technique can be adapted tor use in a stereo vision system.
---
paper_title: Performance of optical flow techniques

paper_content:

The performance of six optical flow techniques is compared, emphasizing measurement accuracy. The most accurate methods are found to be the local differential approaches, where nu is computed explicitly in terms of a locally constant or linear model. Techniques using global smoothness constraints appear to produce visually attractive flow fields, but in general seem to be accurate enough for qualitative use only and insufficient as precursors to the computations of egomotion and 3D structures. It is found that some form of confidence measure/threshold is crucial for all techniques in order to separate the inaccurate from the accurate. Drawbacks of the six techniques are discussed. >
---
paper_title: Moving object detection in spatial domain using background removal techniques -- State-of-art

paper_content:

Identifying moving objects is a critical task for many computer vision applications; it provides a classification of the pixels into either foreground or background. A common approach used to achieve such classification is background removal. Even though there exist numerous of background removal algorithms in the literature, most of them follow a simple flow diagram, passing through four major steps, which are pre-processing, background modelling, foreground de- tection and data validation. In this paper, we survey many existing schemes in the literature of background removal, sur- veying the common pre-processing algorithms used in different situations, presenting different background models, and the most commonly used ways to update such models and how they can be initialized. We also survey how to measure the performance of any moving object detection algorithm, whether the ground truth data is available or not, presenting per- formance metrics commonly used in both cases.
---
paper_title: Efficient adaptive density estimation per image pixel for the task of background subtraction

paper_content:

We analyze the computer vision task of pixel-level background subtraction. We present recursive equations that are used to constantly update the parameters of a Gaussian mixture model and to simultaneously select the appropriate number of components for each pixel. We also present a simple non-parametric adaptive density estimation method. The two methods are compared with each other and with some previously proposed algorithms.
---
paper_title: Efficient Graph-Based Image Segmentation

paper_content:

This paper addresses the problem of segmenting an image into regions. We define a predicate for measuring the evidence for a boundary between two regions using a graph-based representation of the image. We then develop an efficient segmentation algorithm based on this predicate, and show that although this algorithm makes greedy decisions it produces segmentations that satisfy global properties. We apply the algorithm to image segmentation using two different kinds of local neighborhoods in constructing the graph, and illustrate the results with both real and synthetic images. The algorithm runs in time nearly linear in the number of graph edges and is also fast in practice. An important characteristic of the method is its ability to preserve detail in low-variability image regions while ignoring detail in high-variability regions.
---
paper_title: An HMM-based segmentation method for traffic monitoring movies

paper_content:

Shadows of moving objects often obstruct robust visual tracking. We propose an HMM-based segmentation method which classifies in real time each pixel or region into three categories: shadows, foreground, and background objects. In the case of traffic monitoring movies, the effectiveness of the proposed method has been proven through experimental results.
---
paper_title: Determining Optical Flow

paper_content:

Optical flow cannot be computed locally, since only one independent measurement is available from the image sequence at a point, while the flow velocity has two components. A second constraint is needed. A method for finding the optical flow pattern is presented which assumes that the apparent velocity of the brightness pattern varies smoothly almost everywhere in the image. An iterative implementation is shown which successfully computes the optical flow for a number of synthetic image sequences. The algorithm is robust in that it can handle image sequences that are quantified rather coarsely in space and time. It is also insensitive to quantization of brightness levels and additive noise. Examples are included where the assumption of smoothness is violated at singular points or along lines in the image.
---
paper_title: Robust Computer Vision through Kernel Density Estimation

paper_content:

Two new techniques based on nonparametric estimation of probability densities are introduced which improve on the performance of equivalent robust methods currently employed in computer vision. The first technique draws from the projection pursuit paradigm in statistics, and carries out regression M-estimation with a weak dependence on the accuracy of the scale estimate. The second technique exploits the properties of the multivariate adaptive mean shift, and accomplishes the fusion of uncertain measurements arising from an unknown number of sources. As an example, the two techniques are extensively used in an algorithm for the recovery of multiple structures from heavily corrupted data.
---
paper_title: Moving object detection in dynamic scenes based on optical flow and superpixels

paper_content:

Moving object detection under a dynamic background has been a serious challenge in real-time computer vision applications. Global motion compensation approaches, a popular existing technique, aims at compensating the moving background for moving target segmentation. However, it suffers from inaccurate global motion parameters estimation. The paper presents a moving object detection technique that combines TV-L1 optical flow with SLIC superpixel segmentation to characterize moving objects from a dynamic background. SLIC superpixel segmentation can adhere to boundaries of objects, and thus improve the segmentation performance. TV-L1 optical flow implemented on GPU reports competitive smooth flow field with real-time performance. Experimental results on various challenging sequences demonstrate that the proposed approach achieve impressive performance.
---
paper_title: Background modeling and subtraction by codebook construction

paper_content:

We present a new fast algorithm for background modeling and subtraction. Sample background values at each pixel are quantized into codebooks which represent a compressed form of background model for a long image sequence. This allows us to capture structural background variation due to periodic-like motion over a long period of time under limited memory. Our method can handle scenes containing moving backgrounds or illumination variations (shadows and highlights), and it achieves robust detection for compressed videos. We compared our method with other multimode modeling techniques.
---
paper_title: W4: Real-time surveillance of people and their activities

paper_content:

W/sup 4/ is a real time visual surveillance system for detecting and tracking multiple people and monitoring their activities in an outdoor environment. It operates on monocular gray-scale video imagery, or on video imagery from an infrared camera. W/sup 4/ employs a combination of shape analysis and tracking to locate people and their parts (head, hands, feet, torso) and to create models of people's appearance so that they can be tracked through interactions such as occlusions. It can determine whether a foreground region contains multiple people and can segment the region into its constituent people and track them. W/sup 4/ can also determine whether people are carrying objects, and can segment objects from their silhouettes, and construct appearance models for them so they can be identified in subsequent frames. W/sup 4/ can recognize events between people and objects, such as depositing an object, exchanging bags, or removing an object. It runs at 25 Hz for 320/spl times/240 resolution images on a 400 MHz dual-Pentium II PC.
---
paper_title: Modeling, Clustering, and Segmenting Video with Mixtures of Dynamic Textures

paper_content:

A dynamic texture is a spatio-temporal generative model for video, which represents video sequences as observations from a linear dynamical system. This work studies the mixture of dynamic textures, a statistical model for an ensemble of video sequences that is sampled from a finite collection of visual processes, each of which is a dynamic texture. An expectation-maximization (EM) algorithm is derived for learning the parameters of the model, and the model is related to previous works in linear systems, machine learning, time- series clustering, control theory, and computer vision. Through experimentation, it is shown that the mixture of dynamic textures is a suitable representation for both the appearance and dynamics of a variety of visual processes that have traditionally been challenging for computer vision (for example, fire, steam, water, vehicle and pedestrian traffic, and so forth). When compared with state-of-the-art methods in motion segmentation, including both temporal texture methods and traditional representations (for example, optical flow or other localized motion representations), the mixture of dynamic textures achieves superior performance in the problems of clustering and segmenting video of such processes.
---
paper_title: A texture-based method for modeling the background and detecting moving objects

paper_content:

This paper presents a novel and efficient texture-based method for modeling the background and detecting moving objects from a video sequence. Each pixel is modeled as a group of adaptive local binary pattern histograms that are calculated over a circular region around the pixel. The approach provides us with many advantages compared to the state-of-the-art. Experimental results clearly justify our model.
---
paper_title: Background subtraction driven seeds selection for moving objects segmentation and matting

paper_content:

In this paper, we address the difficult task of moving objects segmentation and matting in dynamic scenes. Toward this end, we propose a new automatic way to integrate a background subtraction (BGS) and an alpha matting technique via a heuristic seeds selection scheme. Specifically, our method can be divided into three main steps. First, we use a novel BGS method as attention mechanisms, generating many possible foreground pixels by tuning it for low false-positives and false-negatives as much as possible. Second, a connected components algorithm is used to give the bounding boxes of the labeled foreground pixels. Finally, matting of the object associated to a given bounding box is performed using a heuristic seeds selection scheme. This matting task is guided by top-down knowledge. Experimental results demonstrate the efficiency and effectiveness of our method.
---
paper_title: Complex Background Subtraction by Pursuing Dynamic Spatio-Temporal Models

paper_content:

Although it has been widely discussed in video surveillance, background subtraction is still an open problem in the context of complex scenarios, e.g., dynamic backgrounds, illumination variations, and indistinct foreground objects. To address these challenges, we propose an effective background subtraction method by learning and maintaining an array of dynamic texture models within the spatio-temporal representations. At any location of the scene, we extract a sequence of regular video bricks, i.e., video volumes spanning over both spatial and temporal domain. The background modeling is thus posed as pursuing subspaces within the video bricks while adapting the scene variations. For each sequence of video bricks, we pursue the subspace by employing the auto regressive moving average model that jointly characterizes the appearance consistency and temporal coherence of the observations. During online processing, we incrementally update the subspaces to cope with disturbances from foreground objects and scene changes. In the experiments, we validate the proposed method in several complex scenarios, and show superior performances over other state-of-the-art approaches of background subtraction. The empirical studies of parameter setting and component analysis are presented as well.
---
paper_title: Modeling pixel process with scale invariant local patterns for background subtraction in complex scenes

paper_content:

Background modeling plays an important role in video surveillance, yet in complex scenes it is still a challenging problem. Among many difficulties, problems caused by illumination variations and dynamic backgrounds are the key aspects. In this work, we develop an efficient background subtraction framework to tackle these problems. First, we propose a scale invariant local ternary pattern operator, and show that it is effective for handling illumination variations, especially for moving soft shadows. Second, we propose a pattern kernel density estimation technique to effectively model the probability distribution of local patterns in the pixel process, which utilizes only one single LBP-like pattern instead of histogram as feature. Third, we develop multimodal background models with the above techniques and a multiscale fusion scheme for handling complex dynamic backgrounds. Exhaustive experimental evaluations on complex scenes show that the proposed method is fast and effective, achieving more than 10% improvement in accuracy compared over existing state-of-the-art algorithms.
---
paper_title: Layered Dynamic Textures

paper_content:

A novel video representation, the layered dynamic texture (LDT), is proposed. The LDT is a generative model, which represents a video as a collection of stochastic layers of different appearance and dynamics. Each layer is modeled as a temporal texture sampled from a different linear dynamical system. The LDT model includes these systems, a collection of hidden layer assignment variables (which control the assignment of pixels to layers), and a Markov random field prior on these variables (which encourages smooth segmentations). An EM algorithm is derived for maximum-likelihood estimation of the model parameters from a training video. It is shown that exact inference is intractable, a problem which is addressed by the introduction of two approximate inference procedures: a Gibbs sampler and a computationally efficient variational approximation. The trade-off between the quality of the two approximations and their complexity is studied experimentally. The ability of the LDT to segment videos into layers of coherent appearance and dynamics is also evaluated, on both synthetic and natural videos. These experiments show that the model possesses an ability to group regions of globally homogeneous, but locally heterogeneous, stochastic dynamics currently unparalleled in the literature.
---
paper_title: A texture-based method for detecting moving objects

paper_content:

The detection of moving objects from video frames plays an important and often very critical role in different kinds of machine vision applications including human detection and tracking, traffic monitoring, humanmachine interfaces and military applications, since it usually is one of the first phases in a system architecture. A common way to detect moving objects is background subtraction. In background subtraction, moving objects are detected by comparing each video frame against an existing model of the scene background. In this paper, we propose a novel block-based algorithm for background subtraction. The algorithm is based on the Local Binary Pattern (LBP) texture measure. Each image block is modelled as a group of weighted adaptive LBP histograms. The algorithm operates in real-time under the assumption of a stationary camera with fixed focal length. It can adapt to inherent changes in scene background and can also handle multimodal backgrounds.
---
paper_title: Improving background subtraction using Local Binary Similarity Patterns

paper_content:

Most of the recently published background subtraction methods can still be classified as pixel-based, as most of their analysis is still only done using pixel-by-pixel comparisons. Few others might be regarded as spatial-based (or even spatiotemporal-based) methods, as they take into account the neighborhood of each analyzed pixel. Although the latter types can be viewed as improvements in many cases, most of the methods that have been proposed so far suffer in complexity, processing speed, and/or versatility when compared to their simpler pixel-based counterparts. In this paper, we present an adaptive background subtraction method, derived from the low-cost and highly efficient ViBe method, which uses a spatiotemporal binary similarity descriptor instead of simply relying on pixel intensities as its core component. We then test this method on multiple video sequences and show that by only replacing the core component of a pixel-based method it is possible to dramatically improve its overall performance while keeping memory usage, complexity and speed at acceptable levels for online applications.
---
paper_title: A Biological Hierarchical Model Based Underwater Moving Object Detection

paper_content:

Underwater moving object detection is the key for many underwater computer vision tasks, such as object recognizing, locating, and tracking. Considering the super ability in visual sensing of the underwater habitats, the visual mechanism of aquatic animals is generally regarded as the cue for establishing bionic models which are more adaptive to the underwater environments. However, the low accuracy rate and the absence of the prior knowledge learning limit their adaptation in underwater applications. Aiming to solve the problems originated from the inhomogeneous lumination and the unstable background, the mechanism of the visual information sensing and processing pattern from the eye of frogs are imitated to produce a hierarchical background model for detecting underwater objects. Firstly, the image is segmented into several subblocks. The intensity information is extracted for establishing background model which could roughly identify the object and the background regions. The texture feature of each pixel in the rough object region is further analyzed to generate the object contour precisely. Experimental results demonstrate that the proposed method gives a better performance. Compared to the traditional Gaussian background model, the completeness of the object detection is 97.92% with only 0.94% of the background region that is included in the detection results.
---
paper_title: HMM topology design using maximum likelihood successive state splitting

paper_content:

Abstract Modelling contextual variations of phones is widely accepted as an important aspect of a continuous speech recognition system, and HMM distribution clustering has been sucessfully used to obtain robust models of context through distribution tying. However, as systems move to the challenge of spontaneous speech, temporal variation also becomes important. This paper describes a method fordesigning HMM topologies that learn both temporal and contextual variation, extending previous work on successive state splitting (SSS). The new approach uses a maximum likelihood criterion consistently at each step, overcoming the previous SSS limitation to speaker-dependent training. Initial experiments show both performance gains and training cost reduction over SSS with the reformulated algorithm.
---
paper_title: A Hybrid Color-Based Foreground Object Detection Method for Automated Marine Surveillance

paper_content:

This paper proposes a hybrid foreground object detection method suitable for the marine surveillance applications. Our approach combines an existing foreground object detection method with an image color segmentation technique to improve accuracy. The foreground segmentation method employs a Bayesian decision framework, while the color segmentation part is graph-based and relies on the local variation of edges. We also establish the set of requirements any practical marine surveillance algorithm should fulfill, and show that our method conforms to these requirements. Experiments show good results in the domain of marine surveillance sequences.
---
paper_title: A Probabilistic Background Model for Tracking

paper_content:

A new probabilistic background model based on a Hidden Markov Model is presented. The hidden states of the model enable discrimination between foreground, background and shadow. This model functions as a low level process for a car tracker. A particle filter is employed as a stochastic filter for the car tracker. The use of a particle filter allows the incorporation of the information from the low level process via importance sampling. A novel observation density for the particle filter which models the statistical dependence of neighboring pixels based on a Markov random field is presented. The effectiveness of both the low level process and the observation likelihood are demonstrated.
---
paper_title: Bayesian modeling of dynamic scenes for object detection

paper_content:

Accurate detection of moving objects is an important precursor to stable tracking or recognition. In this paper, we present an object detection scheme that has three innovations over existing approaches. First, the model of the intensities of image pixels as independent random variables is challenged and it is asserted that useful correlation exists in intensities of spatially proximal pixels. This correlation is exploited to sustain high levels of detection accuracy in the presence of dynamic backgrounds. By using a nonparametric density estimation method over a joint domain-range representation of image pixels, multimodal spatial uncertainties and complex dependencies between the domain (location) and range (color) are directly modeled. We propose a model of the background as a single probability density. Second, temporal persistence is proposed as a detection criterion. Unlike previous approaches to object detection which detect objects by building adaptive models of the background, the foregrounds modeled to augment the detection of objects (without explicit tracking) since objects detected in the preceding frame contain substantial evidence for detection in the current frame. Finally, the background and foreground models are used competitively in a MAP-MRF decision framework, stressing spatial context as a condition of detecting interesting objects and the posterior function is maximized efficiently by finding the minimum cut of a capacitated graph. Experimental validation of the proposed method is performed and presented on a diverse set of dynamic scenes.
---
paper_title: A Bayesian computer vision system for modeling human interactions

paper_content:

We describe a real-time computer vision and machine learning system for modeling and recognizing human behaviors in a visual surveillance task. The system deals in particularly with detecting when interactions between people occur and classifying the type of interaction. Examples of interesting interaction behaviors include following another person, altering one's path to meet another, and so forth. Our system combines top-down with bottom-up information in a closed feedback loop, with both components employing a statistical Bayesian approach. We propose and compare two different state-based learning architectures, namely, HMMs and CHMMs for modeling behaviors and interactions. Finally, a synthetic "Alife-style" training system is used to develop flexible prior models for recognizing human interactions. We demonstrate the ability to use these a priori models to accurately classify real human behaviors and interactions with no additional tuning or training.
---
paper_title: Image Segmentation in Video Sequences: A Probabilistic Approach

paper_content:

"Background subtraction" is an old technique for finding moving objects in a video sequence--for example, cars driving on a freeway. The idea is that subtracting the current image from a time-averaged background image will leave only nonstationary objects. It is, however, a crude approximation to the task of classifying each pixel of the current image; it fails with slow-moving objects and does not distinguish shadows from moving objects. The basic idea of this paper is that we can classify each pixel using a model of how that pixel looks when it is part of different classes. We learn a mixture-of-Gaussians classification model for each pixel using an unsupervised technique--an efficient, incremental version of EM. Unlike the standard image-averaging approach, this automatically updates the mixture component for each class according to likelihood of membership; hence slow-moving objects are handled perfectly. Our approach also identifies and eliminates shadows much more effectively than other techniques such as thresholding. Application of this method as part of the Roadwatch traffic surveillance project is expected to result in significant improvements in vehicle identification and tracking.
---
paper_title: Layered Dynamic Textures

paper_content:

A novel video representation, the layered dynamic texture (LDT), is proposed. The LDT is a generative model, which represents a video as a collection of stochastic layers of different appearance and dynamics. Each layer is modeled as a temporal texture sampled from a different linear dynamical system. The LDT model includes these systems, a collection of hidden layer assignment variables (which control the assignment of pixels to layers), and a Markov random field prior on these variables (which encourages smooth segmentations). An EM algorithm is derived for maximum-likelihood estimation of the model parameters from a training video. It is shown that exact inference is intractable, a problem which is addressed by the introduction of two approximate inference procedures: a Gibbs sampler and a computationally efficient variational approximation. The trade-off between the quality of the two approximations and their complexity is studied experimentally. The ability of the LDT to segment videos into layers of coherent appearance and dynamics is also evaluated, on both synthetic and natural videos. These experiments show that the model possesses an ability to group regions of globally homogeneous, but locally heterogeneous, stochastic dynamics currently unparalleled in the literature.
---
paper_title: Soft competitive adaptation: neural network learning algorithms based on fitting statistical mixtures

paper_content:

In this thesis, we consider learning algorithms for neural networks which are based on fitting a mixture probability density to a set of data. ::: We begin with an unsupervised algorithm which is an alternative to the classical winner-take-all competitive algorithms. Rather than updating only the parameters of the "winner" on each case, the parameters of all competitors are updated in proportion to their relative responsibility for the case. Use of such a "soft" competitive algorithm is shown to give better performance than the more traditional algorithms, with little additional cost. ::: We then consider a supervised modular architecture in which a number of simple "expert" networks compete to solve distinct pieces of a large task. A soft competitive mechanism is used to determine how much an expert learns on a case, based on how well the expert performs relative to the other expert networks. At the same time, a separate gating network learns to weight the output of each expert according to a prediction of its relative performance based on the input to the system. Experiments on a number of tasks illustrate that this architecture is capable of uncovering interesting task decompositions and of generalizing better than a single network with small training sets. ::: Finally, we consider learning algorithms in which we assume that the actual output of the network should fall into one of a small number of classes or clusters. The objective of learning is to make the variance of these classes as small as possible. In the classical decision-directed algorithm, we decide that an output belongs to the class it is closest to and minimize the squared distance between the output and the center (mean) of this closest class. In the "soft" version of this algorithm, we minimize the squared distance between the actual output and a weighted average of the means of all of the classes. The weighting factors are the relative probability that the output belongs to each class. This idea may also be used to model the weights of a network, to produce networks which generalize better from small training sets.
---
paper_title: Dynamic texture segmentation

paper_content:

We address the problem of segmenting a sequence of images of natural scenes into disjoint regions that are characterized by constant spatio-temporal statistics. We model the spatio-temporal dynamics in each region by Gauss-Markov models, and infer the model parameters as well as the boundary of the regions in a variational optimization framework. Numerical results demonstrate that - in contrast to purely texture-based segmentation schemes - our method is effective in segmenting regions that differ in their dynamics even when spatial statistics are identical.
---
paper_title: A tutorial on hidden Markov models and selected applications in speech recognition

paper_content:

This tutorial provides an overview of the basic theory of hidden Markov models (HMMs) as originated by L.E. Baum and T. Petrie (1966) and gives practical details on methods of implementation of the theory along with a description of selected applications of the theory to distinct problems in speech recognition. Results from a number of original sources are combined to provide a single source of acquiring the background required to pursue further this area of research. The author first reviews the theory of discrete Markov chains and shows how the concept of hidden states, where the observation is a probabilistic function of the state, can be used effectively. The theory is illustrated with two simple examples, namely coin-tossing, and the classic balls-in-urns system. Three fundamental problems of HMMs are noted and several practical techniques for solving these problems are given. The various types of HMMs that have been studied, including ergodic as well as left-right models, are described. >
---
paper_title: An HMM-based segmentation method for traffic monitoring movies

paper_content:

Shadows of moving objects often obstruct robust visual tracking. We propose an HMM-based segmentation method which classifies in real time each pixel or region into three categories: shadows, foreground, and background objects. In the case of traffic monitoring movies, the effectiveness of the proposed method has been proven through experimental results.
---
paper_title: A dynamic conditional random field model for foreground and shadow segmentation

paper_content:

This paper proposes a dynamic conditional random field (DCRF) model for foreground object and moving shadow segmentation in indoor video scenes. Given an image sequence, temporal dependencies of consecutive segmentation fields and spatial dependencies within each segmentation field are unified by a dynamic probabilistic framework based on the conditional random field (CRF). An efficient approximate filtering algorithm is derived for the DCRF model to recursively estimate the segmentation field from the history of observed images. The foreground and shadow segmentation method integrates both intensity and gradient features. Moreover, models of background, shadow, and gradient information are updated adaptively for nonstationary background processes. Experimental results show that the proposed approach can accurately detect moving objects and their cast shadows even in monocular grayscale video sequences.
---
paper_title: Sparse signal recovery using Markov Random Fields

paper_content:

Compressive Sensing (CS) combines sampling and compression into a single sub-Nyquist linear measurement process for sparse and compressible signals. In this paper, we extend the theory of CS to include signals that are concisely represented in terms of a graphical model. In particular, we use Markov Random Fields (MRFs) to represent sparse signals whose nonzero coefficients are clustered. Our new model-based recovery algorithm, dubbed Lattice Matching Pursuit (LaMP), stably recovers MRF-modeled signals using many fewer measurements and computations than the current state-of-the-art algorithms.
---
paper_title: Segmenting foreground objects from a dynamic textured background via a robust Kalman filter

paper_content:

The algorithm presented aims to segment the foreground objects in video (e.g., people) given time-varying, textured backgrounds. Examples of time-varying backgrounds include waves on water, clouds moving, trees waving in the wind, automobile traffic, moving crowds, escalators, etc. We have developed a novel foreground-background segmentation algorithm that explicitly accounts for the nonstationary nature and clutter-like appearance of many dynamic textures. The dynamic texture is modeled by an autoregressive moving average model (ARMA). A robust Kalman filter algorithm iteratively estimates the intrinsic appearance of the dynamic texture, as well as the regions of the foreground objects. Preliminary experiments with this method have demonstrated promising results.
---
paper_title: Discovery and Segmentation of Activities in Video

paper_content:

Hidden Markov models (HMMs) have become the workhorses of the monitoring and event recognition literature because they bring to time-series analysis the utility of density estimation and the convenience of dynamic time warping. Once trained, the internals of these models are considered opaque; there is no effort to interpret the hidden states. We show that by minimizing the entropy of the joint distribution, an HMM's internal state machine can be made to organize observed activity into meaningful states. This has uses in video monitoring and annotation, low bit-rate coding of scene activity, and detection of anomalous behavior. We demonstrate with models of office activity and outdoor traffic, showing how the framework learns principal modes of activity and patterns of activity change. We then show how this framework can be adapted to infer hidden state from extremely ambiguous images, in particular, inferring 3D body orientation and pose from sequences of low-resolution silhouettes.
---
paper_title: Topology free hidden Markov models: application to background modeling

paper_content:

Hidden Markov models (HMMs) are increasingly being used in computer vision for applications such as: gesture analysis, action recognition from video, and illumination modeling. Their use involves an off-line learning step that is used as a basis for on-line decision making (i.e. a stationarity assumption on the model parameters). But, real-world applications are often non-stationary in nature. This leads to the need for a dynamic mechanism to learn and update the model topology as well as its parameters. This paper presents a new framework for HMM topology and parameter estimation in an online, dynamic fashion. The topology and parameter estimation is posed as a model selection problem with an MDL prior. Online modifications to the topology are made possible by incorporating a state splitting criterion. To demonstrate the potential of the algorithm, the background modeling problem is considered. Theoretical validation and real experiments are presented.
---
paper_title: Joint Motion Segmentation and Background Estimation in Dynamic Scenes

paper_content:

We propose a joint foreground-background mixture model (FBM) that simultaneously performs background estimation and motion segmentation in complex dynamic scenes. Our FBM consist of a set of location-specific dynamic texture (DT) components, for modeling local background motion, and set of global DT components, for modeling consistent foreground motion. We derive an EM algorithm for estimating the parameters of the FBM. We also apply spatial constraints to the FBM using an Markov random field grid, and derive a corresponding variational approximation for inference. Unlike existing approaches to background subtraction, our FBM does not require a manually selected threshold or a separate training video. Unlike existing motion segmentation techniques, our FBM can segment foreground motions over complex background with mixed motions, and detect stopped objects. Since most dynamic scene datasets only contain videos with a single foreground object over a simple background, we develop a new challenging dataset with multiple foreground objects over complex dynamic backgrounds. In experiments, we show that jointly modeling the background and foreground segments with FBM yields significant improvements in accuracy on both background estimation and motion segmentation, compared to state-of-the-art methods.
---
paper_title: A View Of The Em Algorithm That Justifies Incremental, Sparse, And Other Variants

paper_content:

The EM algorithm performs maximum likelihood estimation for data in which some variables are unobserved. We present a function that resembles negative free energy and show that the M step maximizes this function with respect to the model parameters and the E step maximizes it with respect to the distribution over the unobserved variables. From this perspective, it is easy to justify an incremental variant of the EM algorithm in which the distribution for only one of the unobserved variables is recalculated in each E step. This variant is shown empirically to give faster convergence in a mixture estimation problem. A variant of the algorithm that exploits sparse conditional distributions is also described, and a wide range of other variant algorithms are also seen to be possible.
---
paper_title: Moving object detection in spatial domain using background removal techniques -- State-of-art

paper_content:

Identifying moving objects is a critical task for many computer vision applications; it provides a classification of the pixels into either foreground or background. A common approach used to achieve such classification is background removal. Even though there exist numerous of background removal algorithms in the literature, most of them follow a simple flow diagram, passing through four major steps, which are pre-processing, background modelling, foreground de- tection and data validation. In this paper, we survey many existing schemes in the literature of background removal, sur- veying the common pre-processing algorithms used in different situations, presenting different background models, and the most commonly used ways to update such models and how they can be initialized. We also survey how to measure the performance of any moving object detection algorithm, whether the ground truth data is available or not, presenting per- formance metrics commonly used in both cases.
---
paper_title: Modeling, Clustering, and Segmenting Video with Mixtures of Dynamic Textures

paper_content:

A dynamic texture is a spatio-temporal generative model for video, which represents video sequences as observations from a linear dynamical system. This work studies the mixture of dynamic textures, a statistical model for an ensemble of video sequences that is sampled from a finite collection of visual processes, each of which is a dynamic texture. An expectation-maximization (EM) algorithm is derived for learning the parameters of the model, and the model is related to previous works in linear systems, machine learning, time- series clustering, control theory, and computer vision. Through experimentation, it is shown that the mixture of dynamic textures is a suitable representation for both the appearance and dynamics of a variety of visual processes that have traditionally been challenging for computer vision (for example, fire, steam, water, vehicle and pedestrian traffic, and so forth). When compared with state-of-the-art methods in motion segmentation, including both temporal texture methods and traditional representations (for example, optical flow or other localized motion representations), the mixture of dynamic textures achieves superior performance in the problems of clustering and segmenting video of such processes.
---
paper_title: A Self-Organizing Approach to Background Subtraction for Visual Surveillance Applications

paper_content:

Detection of moving objects in video streams is the first relevant step of information extraction in many computer vision applications. Aside from the intrinsic usefulness of being able to segment video streams into moving and background components, detecting moving objects provides a focus of attention for recognition, classification, and activity analysis, making these later steps more efficient. We propose an approach based on self organization through artificial neural networks, widely applied in human image processing systems and more generally in cognitive science. The proposed approach can handle scenes containing moving backgrounds, gradual illumination variations and camouflage, has no bootstrapping limitations, can include into the background model shadows cast by moving objects, and achieves robust detection for different types of videos taken with stationary cameras. We compare our method with other modeling techniques and report experimental results, both in terms of detection accuracy and in terms of processing speed, for color video sequences that represent typical situations critical for video surveillance systems.
---
paper_title: A Model of Saliency-Based Visual Attention for Rapid Scene Analysis

paper_content:

A visual attention system, inspired by the behavior and the neuronal architecture of the early primate visual system, is presented. Multiscale image features are combined into a single topographical saliency map. A dynamical neural network then selects attended locations in order of decreasing saliency. The system breaks down the complex problem of scene understanding by rapidly selecting, in a computationally efficient manner, conspicuous locations to be analyzed in detail. Index Terms—Visual attention, scene analysis, feature extraction, target detection, visual search. ———————— F ————————
---
paper_title: Spatiotemporal Saliency in Dynamic Scenes

paper_content:

A spatiotemporal saliency algorithm based on a center-surround framework is proposed. The algorithm is inspired by biological mechanisms of motion-based perceptual grouping and extends a discriminant formulation of center-surround saliency previously proposed for static imagery. Under this formulation, the saliency of a location is equated to the power of a predefined set of features to discriminate between the visual stimuli in a center and a surround window, centered at that location. The features are spatiotemporal video patches and are modeled as dynamic textures, to achieve a principled joint characterization of the spatial and temporal components of saliency. The combination of discriminant center-surround saliency with the modeling power of dynamic textures yields a robust, versatile, and fully unsupervised spatiotemporal saliency algorithm, applicable to scenes with highly dynamic backgrounds and moving cameras. The related problem of background subtraction is treated as the complement of saliency detection, by classifying nonsalient (with respect to appearance and motion dynamics) points in the visual field as background. The algorithm is tested for background subtraction on challenging sequences, and shown to substantially outperform various state-of-the-art techniques. Quantitatively, its average error rate is almost half that of the closest competitor.
---
paper_title: Background Subtraction Based on Low-Rank and Structured Sparse Decomposition

paper_content:

Low rank and sparse representation based methods, which make few specific assumptions about the background, have recently attracted wide attention in background modeling. With these methods, moving objects in the scene are modeled as pixel-wised sparse outliers. However, in many practical scenarios, the distributions of these moving parts are not truly pixel-wised sparse but structurally sparse. Meanwhile a robust analysis mechanism is required to handle background regions or foreground movements with varying scales. Based on these two observations, we first introduce a class of structured sparsity-inducing norms to model moving objects in videos. In our approach, we regard the observed sequence as being constituted of two terms, a low-rank matrix (background) and a structured sparse outlier matrix (foreground). Next, in virtue of adaptive parameters for dynamic videos, we propose a saliency measurement to dynamically estimate the support of the foreground. Experiments on challenging well known data sets demonstrate that the proposed approach outperforms the state-of-the-art methods and works effectively on a wide range of complex videos.
---
paper_title: Video saliency incorporating spatiotemporal cues and uncertainty weighting

paper_content:

We propose a method to detect visual saliency from video signals by combing both spatial and temporal information and statistical uncertainty measures. The main novelty of the proposed method is twofold. First, separate spatial and temporal saliency maps are generated, where the computation of temporal saliency incorporates a recent psychological study of human visual speed perception, where the perceptual prior probability distribution of the speed of motion is measured through a series of psychovisual experiments. Second, the spatial and temporal saliency maps are merged into one using a spatiotemporally adaptive entropy-based uncertainty weighting approach. Experimental results show that the proposed method significantly outperforms state-of-the-art video saliency detection models.
---
paper_title: Detecting salient motion by accumulating directionally-consistent flow

paper_content:

Motion detection can play an important role in many vision tasks. Yet image motion can arise from "uninteresting" events as well as interesting ones. In this paper, salient motion is defined as motion that is likely to result from a typical surveillance target (e.g., a person or vehicle traveling with a sense of direction through a scene) as opposed to other distracting motions (e.g., the scintillation of specularities on water, the oscillation of vegetation in the wind). We propose an algorithm for detecting this salient motion that is based on intermediate-stage vision integration of optical flow. Empirical results are presented that illustrate the applicability of the proposed methods to real-world video. Unlike many motion detection schemes, no knowledge about expected object size or shape is necessary for rejecting the distracting motion.
---
paper_title: Robust principal component analysis for computer vision

paper_content:

Principal Component Analysis (PCA) has been widely used for the representation of shape, appearance and motion. One drawback of typical PCA methods is that they are least squares estimation techniques and hence fail to account for "outliers" which are common in realistic training sets. In computer vision applications, outliers typically occur within a sample (image) due to pixels that are corrupted by noise, alignment errors, or occlusion. We review previous approaches for making PCA robust to outliers and present a new method that uses an intra-sample outlier process to account for pixel outliers. We develop the theory of Robust Principal Component Analysis (RPCA) and describe a robust M-estimation algorithm for learning linear multi-variate representations of high dimensional data such as images. Quantitative comparisons with traditional PCA and previous robust algorithms illustrate the benefits of RPCA when outliers are present. Details of the algorithm are described and a software implementation is being made publically available.
---
paper_title: A dynamic conditional random field model for foreground and shadow segmentation

paper_content:

This paper proposes a dynamic conditional random field (DCRF) model for foreground object and moving shadow segmentation in indoor video scenes. Given an image sequence, temporal dependencies of consecutive segmentation fields and spatial dependencies within each segmentation field are unified by a dynamic probabilistic framework based on the conditional random field (CRF). An efficient approximate filtering algorithm is derived for the DCRF model to recursively estimate the segmentation field from the history of observed images. The foreground and shadow segmentation method integrates both intensity and gradient features. Moreover, models of background, shadow, and gradient information are updated adaptively for nonstationary background processes. Experimental results show that the proposed approach can accurately detect moving objects and their cast shadows even in monocular grayscale video sequences.
---
paper_title: On the plausibility of the discriminant center-surround hypothesis for visual saliency

paper_content:

It has been suggested that saliency mechanisms play a role in perceptual organization. This work evaluates the plausibility of a recently proposed generic principle for visual saliency: that all saliency decisions are optimal in a decision-theoretic sense. The discriminant saliency hypothesis is combined with the classical assumption that bottom-up saliency is a center-surround process to derive a (decision-theoretic) optimal saliency architecture. Under this architecture, the saliency of each image location is equated to the discriminant power of a set of features with respect to the classification problem that opposes stimuli at center and surround. The optimal saliency detector is derived for various stimulus modalities, including intensity, color, orientation, and motion, and shown to make accurate quantitative predictions of various psychophysics of human saliency for both static and motion stimuli. These include some classical nonlinearities of orientation and motion saliency and a Weber law that governs various types of saliency asymmetries. The discriminant saliency detectors are also applied to various saliency problems of interest in computer vision, including the prediction of human eye fixations on natural scenes, motion-based saliency in the presence of ego-motion, and background subtraction in highly dynamic scenes. In all cases, the discriminant saliency detectors outperform previously proposed methods from both the saliency and the general computer vision literatures.
---
paper_title: Segmentation of infrared image using fuzzy thresholding via local region analysis

paper_content:

According to the characteristic of infrared images, a target extraction method based on fuzzy thresholding is proposed for vehicle target images. A membership function composed of the modified bi-modality and the inverse S adjacency is used. In order to meet the requirement of real time, the bi-modality measure is calculated only in the boundary regions so that the execution time can be greatly reduced. The inverse S adjacency function is used to take full advantage of the position information of the pixels in the reference region. Our method is processed as follows. First, we calculate the membership values consisting of the modified bi-modality and the new adjacency. And then we perform the fuzzy thresholding and the post-processing to extract the precise target from the background. In order to evaluate the performance of our method, the proposed method is compared with other segmentation methods. The results of experiments prove that the presented algorithm is fast and has a good segmentation performance.
---
paper_title: A fuzzy spatial coherence-based approach to background/foreground separation for moving object detection

paper_content:

The detection of moving objects from stationary cameras is usually approached by background subtraction, i.e. by constructing and maintaining an up-to-date model of the background and detecting moving objects as those that deviate from such a model. We adopt a previously proposed approach to background subtraction based on self-organization through artificial neural networks, that has been shown to well cope with several of the well known issues for background maintenance. Here, we propose a spatial coherence variant to such approach to enhance robustness against false detections and formulate a fuzzy model to deal with decision problems typically arising when crisp settings are involved. We show through experimental results and comparisons that higher accuracy values can be reached for color video sequences that represent typical situations critical for moving object detection.
---
paper_title: Motion saliency detection using low-rank and sparse decomposition

paper_content:

Motion saliency detection has an important impact on further video processing tasks, such as video segmentation, object recognition and adaptive compression. Different to image saliency, in videos, moving regions (objects) catch human beings' attention much easier than static ones. Based on this observation, we propose a novel method of motion saliency detection, which makes use of the low-rank and sparse decomposition on video slices along X-T and Y-T planes to achieve the goal, i.e. separating foreground moving objects from backgrounds. In addition, we adopt the spatial information to preserve the completeness of the detected motion objects. In virtue of adaptive threshold selection and efficient noise elimination, the proposed approach is suitable for different video scenes, and robust to low resolution and noisy cases. The experiments demonstrate the performance of our method compared with the state-of-the-art.
---
paper_title: Fuzzy statistical modeling of dynamic backgrounds for moving object detection in infrared videos

paper_content:

Mixture of Gaussians (MOG) is the most popular technique for background modeling and presents some limitations when dynamic changes occur in the scene like camera jitter and movement in the background. Furthermore, the MOG is initialized using a training sequence which may be noisy and/or insufficient to model correctly the background. All these critical situations generate false classification in the foreground detection mask due to the related uncertainty. In this context, we present a background modeling algorithm based on Type-2 Fuzzy Mixture of Gaussians which is particularly suitable for infrared videos. The use of the Type-2 Fuzzy Set Theory allows to take into account the uncertainty. The results using the OTCBVS benchmark/test dataset videos show the robustness of the proposed method in presence of dynamic backgrounds.
---
paper_title: Method for building recognition from FLIR images

paper_content:

Herein, a method for building recognition is presented in forward looking infrared (FLIR) images with clutter background, which is composed of the several sub-procedures. In the first phase, a three-dimensional (3-D) target model is generated and the model features are predicted based on the sensor's perspective relative to the 3-D target model. The second phase of the process, multi-scale structuring elements are generated pertaining to the 3-D target model and flight trajectory. Structuring elements for infrared image is selected by a look-up-table approach based on the parameters of sensor's view, and the use of morphology-based filters can respond to the size and shape of target to suppress the clutter background. In the following process, iterative segmentation for the result image of background suppression is used to obtain regions of interest (ROIs), and features extraction of ROIs and matching retain the ROIs that are closest to predicted features. Lastly, the target is identified by fusing the line features and multi-frame integration. Experiment results show the proposed algorithm can precisely recognize the target from FLIR images with a complicated background.
---
paper_title: Using histograms to detect and track objects in color video

paper_content:

Two methods of detecting and tracking objects in color video are presented. Color and edge histograms are explored as ways to model the background and foreground of a scene. The two types of methods are evaluated to determine their speed, accuracy and robustness. Histogram comparison techniques are used to compute similarity values that aid in identifying regions of interest. Foreground objects are detected and tracked by dividing each video frame into smaller regions (cells) and comparing the histogram of each cell to the background model. Results are presented for video sequences of human activity.
---
paper_title: A Bayesian computer vision system for modeling human interactions

paper_content:

We describe a real-time computer vision and machine learning system for modeling and recognizing human behaviors in a visual surveillance task. The system deals in particularly with detecting when interactions between people occur and classifying the type of interaction. Examples of interesting interaction behaviors include following another person, altering one's path to meet another, and so forth. Our system combines top-down with bottom-up information in a closed feedback loop, with both components employing a statistical Bayesian approach. We propose and compare two different state-based learning architectures, namely, HMMs and CHMMs for modeling behaviors and interactions. Finally, a synthetic "Alife-style" training system is used to develop flexible prior models for recognizing human interactions. We demonstrate the ability to use these a priori models to accurately classify real human behaviors and interactions with no additional tuning or training.
---
paper_title: Efficient background subtraction for real-time tracking in embedded camera networks

paper_content:

Background subtraction is often the first step of many computer vision applications. For a background subtraction method to be useful in embedded camera networks, it must be both accurate and computationally efficient because of the resource constraints on embedded platforms. This makes many traditional background subtraction algorithms unsuitable for embedded platforms because they use complex statistical models to handle subtle illumination changes. These models make them accurate but the computational requirement of these complex models is often too high for embedded platforms. In this paper, we propose a new background subtraction method which is both accurate and computational efficient. The key idea is to use compressive sensing to reduce the dimensionality of the data while retaining most of the information. By using multiple datasets, we show that the accuracy of our proposed background subtraction method is comparable to that of the traditional background subtraction methods. Moreover, real implementation on an embedded camera platform shows that our proposed method is at least 5 times faster, and consumes significantly less energy and memory resources than the conventional approaches. Finally, we demonstrated the feasibility of the proposed method by the implementation and evaluation of an end-to-end real-time embedded camera network target tracking application.
---
paper_title: Independent Component Analysis-Based Background Subtraction for Indoor Surveillance

paper_content:

In video surveillance, detection of moving objects from an image sequence is very important for target tracking, activity recognition, and behavior understanding. Background subtraction is a very popular approach for foreground segmentation in a still scene image. In order to compensate for illumination changes, a background model updating process is generally adopted, and leads to extra computation time. In this paper, we propose a fast background subtraction scheme using independent component analysis (ICA) and, particularly, aims at indoor surveillance for possible applications in home-care and health-care monitoring, where moving and motionless persons must be reliably detected. The proposed method is as computationally fast as the simple image difference method, and yet is highly tolerable to changes in room lighting. The proposed background subtraction scheme involves two stages, one for training and the other for detection. In the training stage, an ICA model that directly measures the statistical independency based on the estimations of joint and marginal probability density functions from relative frequency distributions is first proposed. The proposed ICA model can well separate two highly-correlated images. In the detection stage, the trained de-mixing vector is used to separate the foreground in a scene image with respect to the reference background image. Two sets of indoor examples that involve switching on/off room lights and opening/closing a door are demonstrated in the experiments. The performance of the proposed ICA model for background subtraction is also compared with that of the well-known FastICA algorithm.
---
paper_title: Complex Background Subtraction by Pursuing Dynamic Spatio-Temporal Models

paper_content:

Although it has been widely discussed in video surveillance, background subtraction is still an open problem in the context of complex scenarios, e.g., dynamic backgrounds, illumination variations, and indistinct foreground objects. To address these challenges, we propose an effective background subtraction method by learning and maintaining an array of dynamic texture models within the spatio-temporal representations. At any location of the scene, we extract a sequence of regular video bricks, i.e., video volumes spanning over both spatial and temporal domain. The background modeling is thus posed as pursuing subspaces within the video bricks while adapting the scene variations. For each sequence of video bricks, we pursue the subspace by employing the auto regressive moving average model that jointly characterizes the appearance consistency and temporal coherence of the observations. During online processing, we incrementally update the subspaces to cope with disturbances from foreground objects and scene changes. In the experiments, we validate the proposed method in several complex scenarios, and show superior performances over other state-of-the-art approaches of background subtraction. The empirical studies of parameter setting and component analysis are presented as well.
---
paper_title: Background modeling and subtraction of dynamic scenes

paper_content:

Background modeling and subtraction is a core component in motion analysis. The central idea behind such module is to create a probabilistic representation of the static scene that is compared with the current input to perform subtraction. Such approach is efficient when the scene to be modeled refers to a static structure with limited perturbation. In this paper, we address the problem of modeling dynamic scenes where the assumption of a static background is not valid. Waving trees, beaches, escalators, natural scenes with rain or snow are examples. Inspired by the work proposed by Doretto et al. (2003), we propose an on-line auto-regressive model to capture and predict the behavior of such scenes. Towards detection of events we introduce a new metric that is based on a state-driven comparison between the prediction and the actual frame. Promising results demonstrate the potentials of the proposed framework.
---
paper_title: Wallflower: principles and practice of background maintenance

paper_content:

Background maintenance is a frequent element of video surveillance systems. We develop Wallflower, a three-component system for background maintenance: the pixel-level component performs Wiener filtering to make probabilistic predictions of the expected background; the region-level component fills in homogeneous regions of foreground objects; and the frame-level component detects sudden, global changes in the image and swaps in better approximations of the background. We compare our system with 8 other background subtraction algorithms. Wallflower is shown to outperform previous algorithms by handling a greater set of the difficult situations that can occur. Finally, we analyze the experimental results and propose normative principles for background maintenance.
---
paper_title: Spatiotemporal Saliency Detection and Its Applications in Static and Dynamic Scenes

paper_content:

This paper presents a novel method for detecting salient regions in both images and videos based on a discriminant center-surround hypothesis that the salient region stands out from its surroundings. To this end, our spatiotemporal approach combines the spatial saliency by computing distances between ordinal signatures of edge and color orientations obtained from the center and the surrounding regions and the temporal saliency by simply computing the sum of absolute difference between temporal gradients of the center and the surrounding regions. Our proposed method is computationally efficient, reliable, and simple to implement and thus it can be easily extended to various applications such as image retargeting and moving object extraction. The proposed method has been extensively tested and the results show that the proposed scheme is effective in detecting saliency compared to various state-of-the-art methods.
---
paper_title: Background subtraction for static & moving camera

paper_content:

Background subtraction is one of the most commonly used components in machine vision systems. Despite the numerous algorithms proposed in the literature and used in practical applications, key challenges remain in designing a single system that can handle diverse environmental conditions. In this paper we present Multiple Background Model based Background Subtraction Algorithm as such a candidate. The algorithm was originally designed for handling sudden illumination changes. The new version has been refined with changes at different steps of the process, specifically in terms of selecting optimal color space, clustering of training images for Background Model Bank and parameter for each channel of color space. This has allowed the algorithm's applicability to wide variety of challenges associated with change detection including camera jitter, dynamic background, Intermittent Object Motion, shadows, bad weather, thermal, night videos etc. Comprehensive evaluation demonstrates the superiority of algorithm against state of the art.
---
paper_title: A visual sensing platform for creating a smarter multi-modal marine monitoring network

paper_content:

Demands from various scientific and management communities along with legislative requirements at national and international levels have led to a need for innovative research into large-scale, low-cost, reliable monitoring of our marine and freshwater environments. In this paper we demonstrate the benefits of a multi-modal approach to monitoring and how an in-situ sensor network can be enhanced with the use of contextual image data. We provide an outline of the deployment of a visual sensing system at a busy port and the need for monitoring shipping traffic at the port. Subsequently we present an approach for detecting ships in a challenging image dataset and discuss how this can help to create an intelligent marine monitoring network.
---
paper_title: Spatio-temporal patches for night background modeling by subspace learning

paper_content:

In this paper, a novel background model on spatio-temporal patches is introduced for video surveillance, especially for night outdoor scene, where extreme lighting conditions often cause troubles. The spatio-temporal patch, called brick, is presented to simultaneously capture spatio-temporal information in surveillance video. The set of bricks of a given background patch, under all possible lighting conditions, lies in a low-dimensional subspace, which can be learned by online subspace learning. The proposed method can efficiently model the background and detect the appearance and motion variance caused by foreground. Experimental results on real data show that the proposed method is insensitive to dramatic lighting changes and achieves superior performance to two classical methods.
---
paper_title: Robust principal component analysis for computer vision

paper_content:

Principal Component Analysis (PCA) has been widely used for the representation of shape, appearance and motion. One drawback of typical PCA methods is that they are least squares estimation techniques and hence fail to account for "outliers" which are common in realistic training sets. In computer vision applications, outliers typically occur within a sample (image) due to pixels that are corrupted by noise, alignment errors, or occlusion. We review previous approaches for making PCA robust to outliers and present a new method that uses an intra-sample outlier process to account for pixel outliers. We develop the theory of Robust Principal Component Analysis (RPCA) and describe a robust M-estimation algorithm for learning linear multi-variate representations of high dimensional data such as images. Quantitative comparisons with traditional PCA and previous robust algorithms illustrate the benefits of RPCA when outliers are present. Details of the algorithm are described and a software implementation is being made publically available.
---
paper_title: On the plausibility of the discriminant center-surround hypothesis for visual saliency

paper_content:

It has been suggested that saliency mechanisms play a role in perceptual organization. This work evaluates the plausibility of a recently proposed generic principle for visual saliency: that all saliency decisions are optimal in a decision-theoretic sense. The discriminant saliency hypothesis is combined with the classical assumption that bottom-up saliency is a center-surround process to derive a (decision-theoretic) optimal saliency architecture. Under this architecture, the saliency of each image location is equated to the discriminant power of a set of features with respect to the classification problem that opposes stimuli at center and surround. The optimal saliency detector is derived for various stimulus modalities, including intensity, color, orientation, and motion, and shown to make accurate quantitative predictions of various psychophysics of human saliency for both static and motion stimuli. These include some classical nonlinearities of orientation and motion saliency and a Weber law that governs various types of saliency asymmetries. The discriminant saliency detectors are also applied to various saliency problems of interest in computer vision, including the prediction of human eye fixations on natural scenes, motion-based saliency in the presence of ego-motion, and background subtraction in highly dynamic scenes. In all cases, the discriminant saliency detectors outperform previously proposed methods from both the saliency and the general computer vision literatures.
---
paper_title: Background subtraction under sudden illumination change

paper_content:

In this paper, we propose a Multiple Background Model based Background Subtraction (MB2S) algorithm that is robust against sudden illumination changes in indoor environment. It uses multiple background models of expected illumination changes followed by both pixel and frame based background subtraction on both RGB and YCbCr color spaces. The masks generated after processing these input images are then combined in a framework to classify background and foreground pixels. Evaluation of proposed approach on publicly available test sequences show higher precision and recall than other state-of-the-art algorithms.
---
paper_title: Sparse signal recovery using Markov Random Fields

paper_content:

Compressive Sensing (CS) combines sampling and compression into a single sub-Nyquist linear measurement process for sparse and compressible signals. In this paper, we extend the theory of CS to include signals that are concisely represented in terms of a graphical model. In particular, we use Markov Random Fields (MRFs) to represent sparse signals whose nonzero coefficients are clustered. Our new model-based recovery algorithm, dubbed Lattice Matching Pursuit (LaMP), stably recovers MRF-modeled signals using many fewer measurements and computations than the current state-of-the-art algorithms.
---
paper_title: A Biologically Inspired Vision-Based Approach for Detecting Multiple Moving Objects in Complex Outdoor Scenes

paper_content:

In the human brain, independent components of optical flows from the medial superior temporal area are speculated for motion cognition. Inspired by this hypothesis, a novel approach combining independent component analysis (ICA) with principal component analysis (PCA) is proposed in this paper for multiple moving objects detection in complex scenes—a major real-time challenge as bad weather or dynamic background can seriously influence the results of motion detection. In the proposed approach, by taking advantage of ICA’s capability of separating the statistically independent features from signals, the ICA algorithm is initially employed to analyze the optical flows of consecutive visual image frames. As a result, the optical flows of background and foreground can be approximately separated. Since there are still many disturbances in the foreground optical flows in the complex scene, PCA is then applied to the optical flows of foreground components so that major optical flows corresponding to multiple moving objects can be enhanced effectively and the motions resulted from the changing background and small disturbances are relatively suppressed at the same time. Comparative experimental results with existing popular motion detection methods for challenging imaging sequences demonstrate that our proposed biologically inspired vision-based approach can extract multiple moving objects effectively in a complex scene.
---
paper_title: Segmenting foreground objects from a dynamic textured background via a robust Kalman filter

paper_content:

The algorithm presented aims to segment the foreground objects in video (e.g., people) given time-varying, textured backgrounds. Examples of time-varying backgrounds include waves on water, clouds moving, trees waving in the wind, automobile traffic, moving crowds, escalators, etc. We have developed a novel foreground-background segmentation algorithm that explicitly accounts for the nonstationary nature and clutter-like appearance of many dynamic textures. The dynamic texture is modeled by an autoregressive moving average model (ARMA). A robust Kalman filter algorithm iteratively estimates the intrinsic appearance of the dynamic texture, as well as the regions of the foreground objects. Preliminary experiments with this method have demonstrated promising results.
---
paper_title: Total Variation Regularized RPCA for Irregularly Moving Object Detection Under Dynamic Background

paper_content:

Moving object detection is one of the most fundamental tasks in computer vision. Many classic and contemporary algorithms work well under the assumption that backgrounds are stationary and movements are continuous, but degrade sharply when they are used in a real detection system, mainly due to: 1) the dynamic background (e.g., swaying trees, water ripples and fountains in real scenarios, as well as raindrops and snowflakes in bad weather) and 2) the irregular object movement (like lingering objects). This paper presents a unified framework for addressing the difficulties mentioned above, especially the one caused by irregular object movement. This framework separates dynamic background from moving objects using the spatial continuity of foreground, and detects lingering objects using the temporal continuity of foreground. The proposed framework assumes that the dynamic background is sparser than the moving foreground that has smooth boundary and trajectory. We regard the observed video as being made up of the sum of a low-rank static background, a sparse and smooth foreground, and a sparser dynamic background. To deal with this decomposition, i.e., a constrained minimization problem, the augmented Lagrangian multiplier method is employed with the help of the alternating direction minimizing strategy. Extensive experiments on both simulated and real data demonstrate that our method significantly outperforms the state-of-the-art approaches, especially for the cases with dynamic backgrounds and discontinuous movements.
---
paper_title: Background subtraction based on cooccurrence of image variations

paper_content:

This paper presents a novel background subtraction method for detecting foreground objects in dynamic scenes involving swaying trees and fluttering flags. Most methods proposed so far adjust the permissible range of the background image variations according to the training samples of background images. Thus, the detection sensitivity decreases at those pixels having wide permissible ranges. If we can narrow the ranges by analyzing input images, the detection sensitivity can be improved. For this narrowing, we employ the property that image variations at neighboring image blocks have strong correlation, also known as "cooccurrence". This approach is essentially different from chronological background image updating or morphological postprocessing. Experimental results for real images demonstrate the effectiveness of our method.
---
paper_title: Foreground detection via robust low rank matrix decomposition including spatio-temporal constraint

paper_content:

Foreground detection is the first step in video surveillance system to detect moving objects. Robust Principal Components Analysis (RPCA) shows a nice framework to separate moving objects from the background. The background sequence is then modeled by a low rank subspace that can gradually change over time, while the moving foreground objects constitute the correlated sparse outliers. In this paper, we propose to use a low-rank matrix factorization with IRLS scheme (Iteratively reweighted least squares) and to address in the minimization process the spatial connexity and the temporal sparseness of moving objects (e.g. outliers). Experimental results on the BMC 2012 datasets show the pertinence of the proposed approach.
---
paper_title: A Self-Organizing Approach to Background Subtraction for Visual Surveillance Applications

paper_content:

Detection of moving objects in video streams is the first relevant step of information extraction in many computer vision applications. Aside from the intrinsic usefulness of being able to segment video streams into moving and background components, detecting moving objects provides a focus of attention for recognition, classification, and activity analysis, making these later steps more efficient. We propose an approach based on self organization through artificial neural networks, widely applied in human image processing systems and more generally in cognitive science. The proposed approach can handle scenes containing moving backgrounds, gradual illumination variations and camouflage, has no bootstrapping limitations, can include into the background model shadows cast by moving objects, and achieves robust detection for different types of videos taken with stationary cameras. We compare our method with other modeling techniques and report experimental results, both in terms of detection accuracy and in terms of processing speed, for color video sequences that represent typical situations critical for video surveillance systems.
---
paper_title: A texture-based method for modeling the background and detecting moving objects

paper_content:

This paper presents a novel and efficient texture-based method for modeling the background and detecting moving objects from a video sequence. Each pixel is modeled as a group of adaptive local binary pattern histograms that are calculated over a circular region around the pixel. The approach provides us with many advantages compared to the state-of-the-art. Experimental results clearly justify our model.
---
paper_title: A Bayesian computer vision system for modeling human interactions

paper_content:

We describe a real-time computer vision and machine learning system for modeling and recognizing human behaviors in a visual surveillance task. The system deals in particularly with detecting when interactions between people occur and classifying the type of interaction. Examples of interesting interaction behaviors include following another person, altering one's path to meet another, and so forth. Our system combines top-down with bottom-up information in a closed feedback loop, with both components employing a statistical Bayesian approach. We propose and compare two different state-based learning architectures, namely, HMMs and CHMMs for modeling behaviors and interactions. Finally, a synthetic "Alife-style" training system is used to develop flexible prior models for recognizing human interactions. We demonstrate the ability to use these a priori models to accurately classify real human behaviors and interactions with no additional tuning or training.
---
paper_title: Improving background subtraction using Local Binary Similarity Patterns

paper_content:

Most of the recently published background subtraction methods can still be classified as pixel-based, as most of their analysis is still only done using pixel-by-pixel comparisons. Few others might be regarded as spatial-based (or even spatiotemporal-based) methods, as they take into account the neighborhood of each analyzed pixel. Although the latter types can be viewed as improvements in many cases, most of the methods that have been proposed so far suffer in complexity, processing speed, and/or versatility when compared to their simpler pixel-based counterparts. In this paper, we present an adaptive background subtraction method, derived from the low-cost and highly efficient ViBe method, which uses a spatiotemporal binary similarity descriptor instead of simply relying on pixel intensities as its core component. We then test this method on multiple video sequences and show that by only replacing the core component of a pixel-based method it is possible to dramatically improve its overall performance while keeping memory usage, complexity and speed at acceptable levels for online applications.
---
paper_title: Fuzzy statistical modeling of dynamic backgrounds for moving object detection in infrared videos

paper_content:

Mixture of Gaussians (MOG) is the most popular technique for background modeling and presents some limitations when dynamic changes occur in the scene like camera jitter and movement in the background. Furthermore, the MOG is initialized using a training sequence which may be noisy and/or insufficient to model correctly the background. All these critical situations generate false classification in the foreground detection mask due to the related uncertainty. In this context, we present a background modeling algorithm based on Type-2 Fuzzy Mixture of Gaussians which is particularly suitable for infrared videos. The use of the Type-2 Fuzzy Set Theory allows to take into account the uncertainty. The results using the OTCBVS benchmark/test dataset videos show the robustness of the proposed method in presence of dynamic backgrounds.
---
paper_title: Modeling, Clustering, and Segmenting Video with Mixtures of Dynamic Textures

paper_content:

A dynamic texture is a spatio-temporal generative model for video, which represents video sequences as observations from a linear dynamical system. This work studies the mixture of dynamic textures, a statistical model for an ensemble of video sequences that is sampled from a finite collection of visual processes, each of which is a dynamic texture. An expectation-maximization (EM) algorithm is derived for learning the parameters of the model, and the model is related to previous works in linear systems, machine learning, time- series clustering, control theory, and computer vision. Through experimentation, it is shown that the mixture of dynamic textures is a suitable representation for both the appearance and dynamics of a variety of visual processes that have traditionally been challenging for computer vision (for example, fire, steam, water, vehicle and pedestrian traffic, and so forth). When compared with state-of-the-art methods in motion segmentation, including both temporal texture methods and traditional representations (for example, optical flow or other localized motion representations), the mixture of dynamic textures achieves superior performance in the problems of clustering and segmenting video of such processes.
---
paper_title: Motion Competition: A Variational Approach to Piecewise Parametric Motion Segmentation

paper_content:

We present a novel variational approach for segmenting the image plane into a set of regions of parametric motion on the basis of two consecutive frames from an image sequence. Our model is based on a conditional probability for the spatio-temporal image gradient, given a particular velocity model, and on a geometric prior on the estimated motion field favoring motion boundaries of minimal length. ::: ::: Exploiting the Bayesian framework, we derive a cost functional which depends on parametric motion models for each of a set of regions and on the boundary separating these regions. The resulting functional can be interpreted as an extension of the Mumford-Shah functional from intensity segmentation to motion segmentation. In contrast to most alternative approaches, the problems of segmentation and motion estimation are jointly solved by continuous minimization of a single functional. Minimizing this functional with respect to its dynamic variables results in an eigenvalue problem for the motion parameters and in a gradient descent evolution for the motion discontinuity set. ::: ::: We propose two different representations of this motion boundary: an explicit spline-based implementation which can be applied to the motion-based tracking of a single moving object, and an implicit multiphase level set implementation which allows for the segmentation of an arbitrary number of multiply connected moving objects. ::: ::: Numerical results both for simulated ground truth experiments and for real-world sequences demonstrate the capacity of our approach to segment objects based exclusively on their relative motion.
---
paper_title: Background Subtraction Based on Low-Rank and Structured Sparse Decomposition

paper_content:

Low rank and sparse representation based methods, which make few specific assumptions about the background, have recently attracted wide attention in background modeling. With these methods, moving objects in the scene are modeled as pixel-wised sparse outliers. However, in many practical scenarios, the distributions of these moving parts are not truly pixel-wised sparse but structurally sparse. Meanwhile a robust analysis mechanism is required to handle background regions or foreground movements with varying scales. Based on these two observations, we first introduce a class of structured sparsity-inducing norms to model moving objects in videos. In our approach, we regard the observed sequence as being constituted of two terms, a low-rank matrix (background) and a structured sparse outlier matrix (foreground). Next, in virtue of adaptive parameters for dynamic videos, we propose a saliency measurement to dynamically estimate the support of the foreground. Experiments on challenging well known data sets demonstrate that the proposed approach outperforms the state-of-the-art methods and works effectively on a wide range of complex videos.
---
paper_title: Layered Dynamic Textures

paper_content:

A novel video representation, the layered dynamic texture (LDT), is proposed. The LDT is a generative model, which represents a video as a collection of stochastic layers of different appearance and dynamics. Each layer is modeled as a temporal texture sampled from a different linear dynamical system. The LDT model includes these systems, a collection of hidden layer assignment variables (which control the assignment of pixels to layers), and a Markov random field prior on these variables (which encourages smooth segmentations). An EM algorithm is derived for maximum-likelihood estimation of the model parameters from a training video. It is shown that exact inference is intractable, a problem which is addressed by the introduction of two approximate inference procedures: a Gibbs sampler and a computationally efficient variational approximation. The trade-off between the quality of the two approximations and their complexity is studied experimentally. The ability of the LDT to segment videos into layers of coherent appearance and dynamics is also evaluated, on both synthetic and natural videos. These experiments show that the model possesses an ability to group regions of globally homogeneous, but locally heterogeneous, stochastic dynamics currently unparalleled in the literature.
---
paper_title: Robust principal component analysis?

paper_content:

This paper is about a curious phenomenon. Suppose we have a data matrix, which is the superposition of a low-rank component and a sparse component. Can we recover each component individually? We prove that under some suitable assumptions, it is possible to recover both the low-rank and the sparse components exactly by solving a very convenient convex program called Principal Component Pursuit; among all feasible decompositions, simply minimize a weighted combination of the nuclear norm and of the L1 norm. This suggests the possibility of a principled approach to robust principal component analysis since our methodology and results assert that one can recover the principal components of a data matrix even though a positive fraction of its entries are arbitrarily corrupted. This extends to the situation where a fraction of the entries are missing as well. We discuss an algorithm for solving this optimization problem, and present applications in the area of video surveillance, where our methodology allows for the detection of objects in a cluttered background, and in the area of face recognition, where it offers a principled way of removing shadows and specularities in images of faces.
---
paper_title: Moving Object Detection by Detecting Contiguous Outliers in the Low-Rank Representation

paper_content:

Object detection is a fundamental step for automated video analysis in many vision applications. Object detection in a video is usually performed by object detectors or background subtraction techniques. Often, an object detector requires manually labeled examples to train a binary classifier, while background subtraction needs a training sequence that contains no objects to build a background model. To automate the analysis, object detection without a separate training phase becomes a critical task. People have tried to tackle this task by using motion information. But existing motion-based methods are usually limited when coping with complex scenarios such as nonrigid motion and dynamic background. In this paper, we show that the above challenges can be addressed in a unified framework named DEtecting Contiguous Outliers in the LOw-rank Representation (DECOLOR). This formulation integrates object detection and background learning into a single process of optimization, which can be solved by an alternating algorithm efficiently. We explain the relations between DECOLOR and other sparsity-based methods. Experiments on both simulated data and real sequences demonstrate that DECOLOR outperforms the state-of-the-art approaches and it can work effectively on a wide range of complex scenarios.
---
paper_title: Morphing Active Contours

paper_content:

A method for deforming curves in a given image to a desired position in a second image is introduced in this paper. The algorithm is based on deforming the first image toward the second one via a partial Differential equation, while tracking the deformation of the curves of interest in the first image with an additional, coupled, partial Differential equation. The tracking is performed by projecting the velocities of the first equation into the second one. In contrast with previous PDE based approaches, both the images and the curves on the frames/slices of interest are used for tracking. The technique can be applied to object tracking and sequential segmentation. The topology of the deforming curve can change, without any special topology handling procedures added to the scheme. This permits for example the automatic tracking of scenes where, due to occlusions, the topology of the objects of interest changes from frame to frame. In addition, this work introduces the concept of projecting velocities to obtain systems of coupled partial Differential equations for image analysis applications. We show examples for object tracking and segmentation of electronic microscopy. We also briefly discuss possible uses of this framework iifor three dimensional morphing.
---
paper_title: NLEBS: automatic target detection using a unique nonlinear-enhancement-based system in IR images

paper_content:

A new automatic target detection method for IR images that only requires information about the size of the targets is described. The proposed nonlinear-enhancement-based system (NLEBS) algorithm is based on a nonlinear enhancement paradigm that increases the contrast of the targets with minimal change in the clutter's contrast. The NLEBS employs several stages of processing, each with a different operational purpose. First, the nonlinear enhancement operation is performed by using an iterative procedure. After binarization, segmentation merging causes each local image region to grow by filling in holes. Then segmentation pruning is applied to remove spurious segments. Finally, a heuristic-based metric is employed to validate the possible targets. The performance of the NLEBS was tested with a large set of IR images. The results of these experiments showed a probability of detection greater than 90% and a false-alarm rate of about 1 false alarm per image.
---
paper_title: Segmenting foreground objects from a dynamic textured background via a robust Kalman filter

paper_content:

The algorithm presented aims to segment the foreground objects in video (e.g., people) given time-varying, textured backgrounds. Examples of time-varying backgrounds include waves on water, clouds moving, trees waving in the wind, automobile traffic, moving crowds, escalators, etc. We have developed a novel foreground-background segmentation algorithm that explicitly accounts for the nonstationary nature and clutter-like appearance of many dynamic textures. The dynamic texture is modeled by an autoregressive moving average model (ARMA). A robust Kalman filter algorithm iteratively estimates the intrinsic appearance of the dynamic texture, as well as the regions of the foreground objects. Preliminary experiments with this method have demonstrated promising results.
---
paper_title: Joint Motion Segmentation and Background Estimation in Dynamic Scenes

paper_content:

We propose a joint foreground-background mixture model (FBM) that simultaneously performs background estimation and motion segmentation in complex dynamic scenes. Our FBM consist of a set of location-specific dynamic texture (DT) components, for modeling local background motion, and set of global DT components, for modeling consistent foreground motion. We derive an EM algorithm for estimating the parameters of the FBM. We also apply spatial constraints to the FBM using an Markov random field grid, and derive a corresponding variational approximation for inference. Unlike existing approaches to background subtraction, our FBM does not require a manually selected threshold or a separate training video. Unlike existing motion segmentation techniques, our FBM can segment foreground motions over complex background with mixed motions, and detect stopped objects. Since most dynamic scene datasets only contain videos with a single foreground object over a simple background, we develop a new challenging dataset with multiple foreground objects over complex dynamic backgrounds. In experiments, we show that jointly modeling the background and foreground segments with FBM yields significant improvements in accuracy on both background estimation and motion segmentation, compared to state-of-the-art methods.
---
paper_title: Bayesian modeling of dynamic scenes for object detection

paper_content:

Accurate detection of moving objects is an important precursor to stable tracking or recognition. In this paper, we present an object detection scheme that has three innovations over existing approaches. First, the model of the intensities of image pixels as independent random variables is challenged and it is asserted that useful correlation exists in intensities of spatially proximal pixels. This correlation is exploited to sustain high levels of detection accuracy in the presence of dynamic backgrounds. By using a nonparametric density estimation method over a joint domain-range representation of image pixels, multimodal spatial uncertainties and complex dependencies between the domain (location) and range (color) are directly modeled. We propose a model of the background as a single probability density. Second, temporal persistence is proposed as a detection criterion. Unlike previous approaches to object detection which detect objects by building adaptive models of the background, the foregrounds modeled to augment the detection of objects (without explicit tracking) since objects detected in the preceding frame contain substantial evidence for detection in the current frame. Finally, the background and foreground models are used competitively in a MAP-MRF decision framework, stressing spatial context as a condition of detecting interesting objects and the posterior function is maximized efficiently by finding the minimum cut of a capacitated graph. Experimental validation of the proposed method is performed and presented on a diverse set of dynamic scenes.
---
paper_title: Using adaptive tracking to classify and monitor activities in a site

paper_content:

We describe a vision system that monitors activity in a site over extended periods of time. The system uses a distributed set of sensors to cover the site, and an adaptive tracker detects multiple moving objects in the sensors. Our hypothesis is that motion tracking is sufficient to support a range of computations about site activities. We demonstrate using the tracked motion data to calibrate the distributed sensors, to construct rough site models, to classify detected objects, to learn common patterns of activity for different object classes, and to detect unusual activities.
---
paper_title: An unsupervised, online learning framework for moving object detection

paper_content:

Object detection with a learned classifier has been applied successfully to difficult tasks such as detecting faces and pedestrians. Systems using this approach usually learn the classifier offline with manually labeled training data. We present a framework that learns the classifier online with automatically labeled data for the specific case of detecting moving objects from video. Motion information is used to automatically label training examples collected directly from the live detection task video. An online learner based on the Winnow algorithm incrementally trains a task-specific classifier with these examples. Since learning occurs online and without manual help, it can continue in parallel with detection and adapt the classifier over time. The framework is demonstrated on a person detection task for an office corridor scene. In this task, we use background subtraction to automatically label training examples. After the initial manual effort of implementing the labeling method, the framework runs by itself on the scene video stream to gradually train an accurate detector.
---
paper_title: Semi-Supervised On-line Boosting for Robust Tracking

paper_content:

Recently, on-line adaptation of binary classifiers for tracking have been investigated. On-line learning allows for simple classifiers since only the current view of the object from its surrounding background needs to be discriminiated. However, on-line adaption faces one key problem: Each update of the tracker may introduce an error which, finally, can lead to tracking failure (drifting). The contribution of this paper is a novel on-line semi-supervised boosting method which significantly alleviates the drifting problem in tracking applications. This allows to limit the drifting problem while still staying adaptive to appearance changes. The main idea is to formulate the update process in a semi-supervised fashion as combined decision of a given prior and an on-line classifier. This comes without any parameter tuning. In the experiments, we demonstrate real-time tracking of our SemiBoost tracker on several challenging test sequences where our tracker outperforms other on-line tracking methods.
---
paper_title: Incremental Learning for Robust Visual Tracking

paper_content:

Visual tracking, in essence, deals with non-stationary image streams that change over time. While most existing algorithms are able to track objects well in controlled environments, they usually fail in the presence of significant variation of the object's appearance or surrounding illumination. One reason for such failures is that many algorithms employ fixed appearance models of the target. Such models are trained using only appearance data available before tracking begins, which in practice limits the range of appearances that are modeled, and ignores the large volume of information (such as shape changes or specific lighting conditions) that becomes available during tracking. In this paper, we present a tracking method that incrementally learns a low-dimensional subspace representation, efficiently adapting online to changes in the appearance of the target. The model update, based on incremental algorithms for principal component analysis, includes two important features: a method for correctly updating the sample mean, and a forgetting factor to ensure less modeling power is expended fitting older observations. Both of these features contribute measurably to improving overall tracking performance. Numerous experiments demonstrate the effectiveness of the proposed tracking algorithm in indoor and outdoor environments where the target objects undergo large changes in pose, scale, and illumination.
---
paper_title: A Boosted Particle Filter: Multitarget Detection and Tracking

paper_content:

The problem of tracking a varying number of non-rigid objects has two major difficulties. First, the observation models and target distributions can be highly non-linear and non-Gaussian. Second, the presence of a large, varying number of objects creates complex interactions with overlap and ambiguities. To surmount these difficulties, we introduce a vision system that is capable of learning, detecting and tracking the objects of interest. The system is demonstrated in the context of tracking hockey players using video sequences. Our approach combines the strengths of two successful algorithms: mixture particle filters and Adaboost. The mixture particle filter [17] is ideally suited to multi-target tracking as it assigns a mixture component to each player. The crucial design issues in mixture particle filters are the choice of the proposal distribution and the treatment of objects leaving and entering the scene. Here, we construct the proposal distribution using a mixture model that incorporates information from the dynamic models of each player and the detection hypotheses generated by Adaboost. The learned Adaboost proposal distribution allows us to quickly detect players entering the scene, while the filtering process enables us to keep track of the individual players. The result of interleaving Adaboost with mixture particle filters is a simple, yet powerful and fully automatic multiple object tracking system.
---
paper_title: EigenTracking: Robust Matching and Tracking of Articulated Objects Using a View-Based Representation

paper_content:

This paper describes an approach for tracking rigid and articulated objects using a view-based representation. The approach builds on and extends work on eigenspace representations, robust estimation techniques, and parameterized optical flow estimation. First, we note that the least-squares image reconstruction of standard eigenspace techniques has a number of problems and we reformulate the reconstruction problem as one of robust estimation. Second we define a “subspace constancy assumption” that allows us to exploit techniques for parameterized optical flow estimation to simultaneously solve for the view of an object and the affine transformation between the eigenspace and the image. To account for large affine transformations between the eigenspace and the image we define a multi-scale eigenspace representation and a coarse-to-fine matching strategy. Finally, we use these techniques to track objects over long image sequences in which the objects simultaneously undergo both affine image motions and changes of view. In particular we use this “EigenTracking” technique to track and recognize the gestures of a moving hand.
---
paper_title: Motion Competition: A Variational Approach to Piecewise Parametric Motion Segmentation

paper_content:

We present a novel variational approach for segmenting the image plane into a set of regions of parametric motion on the basis of two consecutive frames from an image sequence. Our model is based on a conditional probability for the spatio-temporal image gradient, given a particular velocity model, and on a geometric prior on the estimated motion field favoring motion boundaries of minimal length. ::: ::: Exploiting the Bayesian framework, we derive a cost functional which depends on parametric motion models for each of a set of regions and on the boundary separating these regions. The resulting functional can be interpreted as an extension of the Mumford-Shah functional from intensity segmentation to motion segmentation. In contrast to most alternative approaches, the problems of segmentation and motion estimation are jointly solved by continuous minimization of a single functional. Minimizing this functional with respect to its dynamic variables results in an eigenvalue problem for the motion parameters and in a gradient descent evolution for the motion discontinuity set. ::: ::: We propose two different representations of this motion boundary: an explicit spline-based implementation which can be applied to the motion-based tracking of a single moving object, and an implicit multiphase level set implementation which allows for the segmentation of an arbitrary number of multiply connected moving objects. ::: ::: Numerical results both for simulated ground truth experiments and for real-world sequences demonstrate the capacity of our approach to segment objects based exclusively on their relative motion.
---
paper_title: Motion-based background subtraction using adaptive kernel density estimation

paper_content:

Background modeling is an important component of many vision systems. Existing work in the area has mostly addressed scenes that consist of static or quasi-static structures. When the scene exhibits a persistent dynamic behavior in time, such an assumption is violated and detection performance deteriorates. In this paper, we propose a new method for the modeling and subtraction of such scenes. Towards the modeling of the dynamic characteristics, optical flow is computed and utilized as a feature in a higher dimensional space. Inherent ambiguities in the computation of features are addressed by using a data-dependent bandwidth for density estimation using kernels. Extensive experiments demonstrate the utility and performance of the proposed approach.
---
paper_title: A Unified Algebraic Approach to 2-D and 3-D Motion Segmentation

paper_content:

We present an analytic solution to the problem of estimating multiple 2-D and 3-D motion models from two-view correspondences or optical flow. The key to our approach is to view the estimation of multiple motion models as the estimation of a single multibody motion model. This is possible thanks to two important algebraic facts. First, we show that all the image measurements, regardless of their associated motion model, can be fit with a real or complex polynomial. Second, we show that the parameters of the motion model associated with an image measurement can be obtained from the derivatives of the polynomial at the measurement. This leads to a novel motion segmentation algorithm that applies to most of the two-view motion models adopted in computer vision. Our experiments show that the proposed algorithm outperforms existing algebraic methods in terms of efficiency and robustness, and provides a good initialization for iterative techniques, such as EM, which is strongly dependent on correct initialization.
---
paper_title: Piecewise-Smooth Dense Optical Flow via Level Sets

paper_content:

We propose a new algorithm for dense optical flow computation. Dense optical flow schemes are challenged by the presence of motion discontinuities. In state of the art optical flow methods, over-smoothing of flow discontinuities accounts for most of the error. A breakthrough in the performance of optical flow computation has recently been achieved by Brox et~al. Our algorithm embeds their functional within a two phase active contour segmentation framework. Piecewise-smooth flow fields are accommodated and flow boundaries are crisp. Experimental results show the superiority of our algorithm with respect to alternative techniques. We also study a special case of optical flow computation, in which the camera is static. In this case we utilize a known background image to separate the moving elements in the sequence from the static elements. Tests with challenging real world sequences demonstrate the performance gains made possible by incorporating the static camera assumption in our algorithm.
---
paper_title: Detecting salient motion by accumulating directionally-consistent flow

paper_content:

Motion detection can play an important role in many vision tasks. Yet image motion can arise from "uninteresting" events as well as interesting ones. In this paper, salient motion is defined as motion that is likely to result from a typical surveillance target (e.g., a person or vehicle traveling with a sense of direction through a scene) as opposed to other distracting motions (e.g., the scintillation of specularities on water, the oscillation of vegetation in the wind). We propose an algorithm for detecting this salient motion that is based on intermediate-stage vision integration of optical flow. Empirical results are presented that illustrate the applicability of the proposed methods to real-world video. Unlike many motion detection schemes, no knowledge about expected object size or shape is necessary for rejecting the distracting motion.
---
paper_title: The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields

paper_content:

Most approaches for estimating optical flow assume that, within a finite image region, only a single motion is present. Thissingle motion assumptionis violated in common situations involving transparency, depth discontinuities, independently moving objects, shadows, and specular reflections. To robustly estimate optical flow, the single motion assumption must be relaxed. This paper presents a framework based onrobust estimationthat addresses violations of the brightness constancy and spatial smoothness assumptions caused by multiple motions. We show how therobust estimation frameworkcan be applied to standard formulations of the optical flow problem thus reducing their sensitivity to violations of their underlying assumptions. The approach has been applied to three standard techniques for recovering optical flow: area-based regression, correlation, and regularization with motion discontinuities. This paper focuses on the recovery of multiple parametric motion models within a region, as well as the recovery of piecewise-smooth flow fields, and provides examples with natural and synthetic image sequences.
---
paper_title: Background models for tracking objects in water

paper_content:

This paper presents a novel background analysis technique to enable robust tracking of objects in water- based scenarios. Current pixel-wise statistical background models support automatic change detection in many outdoor situations, but are limited to background changes which can be modeled via a set of per-pixel spatially uncorrelated processes. In water-based scenarios, waves caused by wind or by moving vessels (wakes) form highly correlated moving patterns that confuse traditional background analysis models. In this work we introduce a framework that explicitly models this type of background variation. The framework combines the output of a statistical background model with localized optical flow analysis to produce two motion maps. In the final stage we apply object-level fusion to filter out moving regions that are most likely caused by wave clutter. A tracking algorithm can now handle the resulting set of objects.
---
paper_title: A Benchmark for the Comparison of 3-D Motion Segmentation Algorithms

paper_content:

Over the past few years, several methods for segmenting a scene containing multiple rigidly moving objects have been proposed. However, most existing methods have been tested on a handful of sequences only, and each method has been often tested on a different set of sequences. Therefore, the comparison of different methods has been fairly limited. In this paper, we compare four 3D motion segmentation algorithms for affine cameras on a benchmark of 155 motion sequences of checkerboard, traffic, and articulated scenes.
---
paper_title: A Unified Algebraic Approach to 2-D and 3-D Motion Segmentation

paper_content:

We present an analytic solution to the problem of estimating multiple 2-D and 3-D motion models from two-view correspondences or optical flow. The key to our approach is to view the estimation of multiple motion models as the estimation of a single multibody motion model. This is possible thanks to two important algebraic facts. First, we show that all the image measurements, regardless of their associated motion model, can be fit with a real or complex polynomial. Second, we show that the parameters of the motion model associated with an image measurement can be obtained from the derivatives of the polynomial at the measurement. This leads to a novel motion segmentation algorithm that applies to most of the two-view motion models adopted in computer vision. Our experiments show that the proposed algorithm outperforms existing algebraic methods in terms of efficiency and robustness, and provides a good initialization for iterative techniques, such as EM, which is strongly dependent on correct initialization.
---
paper_title: Object segmentation in video: A hierarchical variational approach for turning point trajectories into dense regions

paper_content:

Point trajectories have emerged as a powerful means to obtain high quality and fully unsupervised segmentation of objects in video shots. They can exploit the long term motion difference between objects, but they tend to be sparse due to computational reasons and the difficulty in estimating motion in homogeneous areas. In this paper we introduce a variational method to obtain dense segmentations from such sparse trajectory clusters. Information is propagated with a hierarchical, nonlinear diffusion process that runs in the continuous domain but takes superpixels into account. We show that this process raises the density from 3% to 100% and even increases the average precision of labels.
---
paper_title: Object segmentation by long term analysis of point trajectories

paper_content:

Unsupervised learning requires a grouping step that defines which data belong together. A natural way of grouping in images is the segmentation of objects or parts of objects. While pure bottom-up segmentation from static cues is well known to be ambiguous at the object level, the story changes as soon as objects move. In this paper, we present a method that uses long term point trajectories based on dense optical flow. Defining pair-wise distances between these trajectories allows to cluster them, which results in temporally consistent segmentations of moving objects in a video shot. In contrast to multi-body factorization, points and even whole objects may appear or disappear during the shot. We provide a benchmark dataset and an evaluation method for this so far uncovered setting.
---
paper_title: Moving Object Detection by Detecting Contiguous Outliers in the Low-Rank Representation

paper_content:

Object detection is a fundamental step for automated video analysis in many vision applications. Object detection in a video is usually performed by object detectors or background subtraction techniques. Often, an object detector requires manually labeled examples to train a binary classifier, while background subtraction needs a training sequence that contains no objects to build a background model. To automate the analysis, object detection without a separate training phase becomes a critical task. People have tried to tackle this task by using motion information. But existing motion-based methods are usually limited when coping with complex scenarios such as nonrigid motion and dynamic background. In this paper, we show that the above challenges can be addressed in a unified framework named DEtecting Contiguous Outliers in the LOw-rank Representation (DECOLOR). This formulation integrates object detection and background learning into a single process of optimization, which can be solved by an alternating algorithm efficiently. We explain the relations between DECOLOR and other sparsity-based methods. Experiments on both simulated data and real sequences demonstrate that DECOLOR outperforms the state-of-the-art approaches and it can work effectively on a wide range of complex scenarios.
---
paper_title: Joint Motion Segmentation and Background Estimation in Dynamic Scenes

paper_content:

We propose a joint foreground-background mixture model (FBM) that simultaneously performs background estimation and motion segmentation in complex dynamic scenes. Our FBM consist of a set of location-specific dynamic texture (DT) components, for modeling local background motion, and set of global DT components, for modeling consistent foreground motion. We derive an EM algorithm for estimating the parameters of the FBM. We also apply spatial constraints to the FBM using an Markov random field grid, and derive a corresponding variational approximation for inference. Unlike existing approaches to background subtraction, our FBM does not require a manually selected threshold or a separate training video. Unlike existing motion segmentation techniques, our FBM can segment foreground motions over complex background with mixed motions, and detect stopped objects. Since most dynamic scene datasets only contain videos with a single foreground object over a simple background, we develop a new challenging dataset with multiple foreground objects over complex dynamic backgrounds. In experiments, we show that jointly modeling the background and foreground segments with FBM yields significant improvements in accuracy on both background estimation and motion segmentation, compared to state-of-the-art methods.
---
paper_title: Forward-Backward Error: Automatic Detection of Tracking Failures

paper_content:

This paper proposes a novel method for tracking failure detection. The detection is based on the Forward-Backward error, i.e. the tracking is performed forward and backward in time and the discrepancies between these two trajectories are measured. We demonstrate that the proposed error enables reliable detection of tracking failures and selection of reliable trajectories in video sequences. We demonstrate that the approach is complementary to commonly used normalized cross-correlation (NCC). Based on the error, we propose a novel object tracker called Median Flow. State-of-the-art performance is achieved on challenging benchmark video sequences which include non-rigid objects.
---
paper_title: Tracking-Learning-Detection

paper_content:

This paper investigates long-term tracking of unknown objects in a video stream. The object is defined by its location and extent in a single frame. In every frame that follows, the task is to determine the object's location and extent or indicate that the object is not present. We propose a novel tracking framework (TLD) that explicitly decomposes the long-term tracking task into tracking, learning, and detection. The tracker follows the object from frame to frame. The detector localizes all appearances that have been observed so far and corrects the tracker if necessary. The learning estimates the detector's errors and updates it to avoid these errors in the future. We study how to identify the detector's errors and learn from them. We develop a novel learning method (P-N learning) which estimates the errors by a pair of “experts”: (1) P-expert estimates missed detections, and (2) N-expert estimates false alarms. The learning process is modeled as a discrete dynamical system and the conditions under which the learning guarantees improvement are found. We describe our real-time implementation of the TLD framework and the P-N learning. We carry out an extensive quantitative evaluation which shows a significant improvement over state-of-the-art approaches.
---
paper_title: Adaptive Color Attributes for Real-Time Visual Tracking

paper_content:

Visual tracking is a challenging problem in computer vision. Most state-of-the-art visual trackers either rely on luminance information or use simple color representations for image description. Contrary to visual tracking, for object recognition and detection, sophisticated color features when combined with luminance have shown to provide excellent performance. Due to the complexity of the tracking problem, the desired color feature should be computationally efficient, and possess a certain amount of photometric invariance while maintaining high discriminative power. This paper investigates the contribution of color in a tracking-by-detection framework. Our results suggest that color attributes provides superior performance for visual tracking. We further propose an adaptive low-dimensional variant of color attributes. Both quantitative and attribute-based evaluations are performed on 41 challenging benchmark color sequences. The proposed approach improves the baseline intensity-based tracker by 24 % in median distance precision. Furthermore, we show that our approach outperforms state-of-the-art tracking methods while running at more than 100 frames per second.
---
paper_title: Automatic classification of ships from infrared (FLIR) images

paper_content:

The aim of the research presented in this paper is to find out whether automatic classification of ships from Forward Looking InfraRed (FLIR) images is feasible in maritime patrol aircraft. An image processing system has been developed for this task. It includes iterative shading correction and a top hat filter for the detection of the ship. It uses a segmentation algorithm based on the gray value distribution of the waves and the Hough transform to locate the waterline of the ship. A model has been developed to relate the size of the ship and the angle between waterline and horizon in image coordinates, to the real-life size and aspect angle of the ship. The model uses the camera elevation and distance to the ship. A data set was used consisting of two civil ships and four different frigates under different aspect angles and distances. From each of these ship images, 32 features were calculated, among which are the apparent size, the location of the hot spot and of the superstructures of the ship, and moment invariant functions. All features were used in feature selection processing using both the Mahalanobis and nearest neighbor (NN) criteria in forward, backward, and branch & bound feature selection procedures, to find the most significant features. Classification has been performed using a k-NN, a linear, and a quadratic classifier. In particular, using the 1-NN classifier, good results were achieved using a two-step classification algorithm.
---
paper_title: Detection and tracking of moving objects in a maritime environment using level set with shape priors

paper_content:

Over the years, maritime surveillance has become increasingly important due to the recurrence of piracy. While surveillance has traditionally been a manual task using crew members in lookout positions on parts of the ship, much work is being done to automate this task using digital cameras coupled with a computer that uses image processing techniques that intelligently track object in the maritime environment. One such technique is level set segmentation which evolves a contour to objects of interest in a given image. This method works well but gives incorrect segmentation results when a target object is corrupted in the image. This paper explores the possibility of factoring in prior knowledge of a ship’s shape into level set segmentation to improve results, a concept that is unaddressed in maritime surveillance problem. It is shown that the developed video tracking system outperforms level set-based systems that do not use prior shape knowledge, working well even where these systems fail.
---
paper_title: Argos - a Video Surveillance System for boat Traffic Monitoring in Venice

paper_content:

Visual surveillance in dynamic scenes is currently one of the most active research topics in computer vision, many existing applications are available. However, difficulties in realizing effective video surveillance systems that are robust to the many different conditions that arise in real environments, make the actual deployment of such systems very challenging. In this article, we present a real, unique and pioneer video surveillance system for boat traffic monitoring, ARGOS. The system runs continuously 24 hours a day, 7 days a week, day and night in the city of Venice (Italy) since 2007 and it is able to build a reliable background model of the water channel and to track the boats navigating the channel with good accuracy in real-time. A significant experimental evaluation, reported in this article, has been performed in order to assess the real performance of the system.
---
paper_title: A novel framework for making dominant point detection methods non-parametric

paper_content:

Most dominant point detection methods require heuristically chosen control parameters. One of the commonly used control parameter is maximum deviation. This paper uses a theoretical bound of the maximum deviation of pixels obtained by digitization of a line segment for constructing a general framework to make most dominant point detection methods non-parametric. The derived analytical bound of the maximum deviation can be used as a natural bench mark for the line fitting algorithms and thus dominant point detection methods can be made parameter-independent and non-heuristic. Most methods can easily incorporate the bound. This is demonstrated using three categorically different dominant point detection methods. Such non-parametric approach retains the characteristics of the digital curve while providing good fitting performance and compression ratio for all the three methods using a variety of digital, non-digital, and noisy curves.
---
paper_title: A Novel Hierarchical Method of Ship Detection from Spaceborne Optical Image Based on Shape and Texture Features

paper_content:

Ship detection from remote sensing imagery is very important, with a wide array of applications in areas such as fishery management, vessel traffic services, and naval warfare. This paper focuses on the issue of ship detection from spaceborne optical images (SDSOI). Although advantages of synthetic-aperture radar (SAR) result in that most of current ship detection approaches are based on SAR images, disadvantages of SAR still exist, such as the limited number of SAR sensors, the relatively long revisit cycle, and the relatively lower resolution. With the increasing number of and the resulting improvement in continuous coverage of the optical sensors, SDSOI can partly overcome the shortcomings of SAR-based approaches and should be investigated to help satisfy the requirements of real-time ship monitoring. In SDSOI, several factors such as clouds, ocean waves, and small islands affect the performance of ship detection. This paper proposes a novel hierarchical complete and operational SDSOI approach based on shape and texture features, which is considered a sequential coarse-to-fine elimination process of false alarms. First, simple shape analysis is adopted to eliminate evident false candidates generated by image segmentation with global and local information and to extract ship candidates with missing alarms as low as possible. Second, a novel semisupervised hierarchical classification approach based on various features is presented to distinguish between ships and nonships to remove most false alarms. Besides a complete and operational SDSOI approach, the other contributions of our approach include the following three aspects: 1) it classifies ship candidates by using their class probability distributions rather than the direct extracted features; 2) the relevant classes are automatically built by the samples' appearances and their feature attribute in a semisupervised mode; and 3) besides commonly used shape and texture features, a new texture operator, i.e., local multiple patterns, is introduced to enhance the representation ability of the feature set in feature extraction. Experimental results of SDSOI on a large image set captured by optical sensors from multiple satellites show that our approach is effective in distinguishing between ships and nonships, and obtains a satisfactory ship detection performance.
---
paper_title: Ship detection by different data selection templates and multilayer perceptrons from incoherent maritime radar data

paper_content:

This study presents a novel way for detecting ships in sea clutter. For this purpose, the information contained in the Radar images obtained by an incoherent X-band maritime Radar is used. The ship detection is solved by feedforward artificial neural networks, such as the multilayer perceptrons (MLPs). In a first approach, the MLP processes the information extracted from the Radar images using the commonly used horizontal and vertical selection templates. But, if a suitable combination of these selection templates is done, better detection performances are achieved. So, two improved selection templates are proposed, which are based on cross and plus shapes. All these templates are also applied in a commonly used detector taken as reference, the CA-CFAR detector. Its performance is compared with the one achieved by the proposed detector. This comparison shows how the MLP-based detector outperforms the CA-CFAR detector in all the cases under study. The results are presented in terms of objective (probabilities of false alarm and detection) and subjective estimations of their performances. The improved MLP-based detector also presents low computational cost and high robustness in its performance against changes in the sea conditions and ship properties.
---
paper_title: Persistent maritime surveillance using multi-sensor feature association and classification

paper_content:

In maritime operational scenarios, such as smuggling, piracy, or terrorist threats, it is not only relevant who or what an observed object is, but also where it is now and in the past in relation to other (geographical) objects. In situation and impact assessment, this information is used to determine whether an object is a threat. Single platform (ship, harbor) or single sensor information will not provide all this information. The work presented in this paper focuses on the sensor and object levels that provide a description of currently observed objects to situation assessment. For use of information of objects at higher information levels, it is necessary to have not only a good description of observed objects at this moment, but also from its past. Therefore, currently observed objects have to be linked to previous occurrences. Kinematic features, as used in tracking, are of limited use, as uncertainties over longer time intervals are so large that no unique associations can be made. Features extracted from different sensors (e.g., ESM, EO/IR) can be used for both association and classification. Features and classifications are used to associate current objects to previous object descriptions, allowing objects to be described better, and provide position history. In this paper a description of a high level architecture in which such a multi-sensor association is used is described. Results of an assessment of the usability of several features from ESM (from spectrum), EO and IR (shape, contour, keypoints) data for association and classification are shown. © 2012 Copyright Society of Photo-Optical Instrumentation Engineers (SPIE).
---
paper_title: Maritime Surveillance : Tracking Ships inside a Dynamic Background Using a Fast Level-Set

paper_content:

Research highlights? Existing vision-based tracking methods are not suitable for the maritime domain. ? We derive a suitable tracking method by combining and modifying existing methods. ? Our method can track tiny targets. ? Our method is validated on several test sequences and two live field trials. Surveillance in a maritime environment is indispensable in the fight against a wide range of criminal activities, including pirate attacks, unlicensed fishing trailers and human trafficking. Computer vision systems can be a useful aid in the law enforcement process, by for example tracking and identifying moving vessels on the ocean. However, the maritime domain poses many challenges for the design of an effective maritime surveillance system. One such challenge is the tracking of moving vessels in the presence of a moving dynamic background (the ocean). We present techniques that address this particular problem. We use a background subtraction method and employ a real-time approximation of level-set-based curve evolution to demarcate the outline of moving vessels in the ocean. We report promising results on both small and large vessels, based on two field trials.
---
paper_title: AUTOMATIC MARITIME SURVEILLANCE WITH VISUAL TARGET DETECTION

paper_content:

In this paper an automatic maritime surveillance system is presented. Boat detection is performed by means of an Haar-like classifier in order to obtain robustness with respect to targets having very different size, reflections and wakes on the water surface, and apparently motionless boats anchored off the coast. Detection results are filtered over the time in order to reduce the false alarm rate. Experimental results show the effectiveness of the approach with different light conditions and camera positions. The system is able to provide the user a global view adding a visual dimension to AIS data.
---
paper_title: Adaptive maritime video surveillance

paper_content:

Maritime assets such as ports, harbors, and vessels are vulnerable to a variety of near-shore threats such as small-boat attacks. Currently, such vulnerabilities are addressed predominantly by watchstanders and manual video surveillance, which is manpower intensive. Automatic maritime video surveillance techniques are being introduced to reduce manpower costs, but they have limited functionality and performance. For example, they only detect simple events such as perimeter breaches and cannot predict emerging threats. They also generate too many false alerts and cannot explain their reasoning. To overcome these limitations, we are developing the Maritime Activity Analysis Workbench (MAAW), which will be a mixed-initiative real-time maritime video surveillance tool that uses an integrated supervised machine learning approach to label independent and coordinated maritime activities. It uses the same information to predict anomalous behavior and explain its reasoning; this is an important capability for watchstander training and for collecting performance feedback. In this paper, we describe MAAW's functional architecture, which includes the following pipeline of components: (1) a video acquisition and preprocessing component that detects and tracks vessels in video images, (2) a vessel categorization and activity labeling component that uses standard and relational supervised machine learning methods to label maritime activities, and (3) an ontology-guided vessel and maritime activity annotator to enable subject matter experts (e.g., watchstanders) to provide feedback and supervision to the system. We report our findings from a preliminary system evaluation on river traffic video.
---
paper_title: Ship recognition for improved persistent tracking with descriptor localization and compact representations

paper_content:

For maritime situational awareness, it is important to identify currently observed ships as earlier encounters. For example, past location and behavior analysis are useful to determine whether a ship is of interest in case of piracy and smuggling. It is beneficial to verify this with cameras at a distance, to avoid the costs of bringing an own asset closer to the ship. The focus of this paper is on ship recognition from electro-optical imagery. The main contribution is an analysis of the effect of using the combination of descriptor localization and compact representations. An evaluation is performed to assess the usefulness in persistent tracking, especially for larger intervals (i.e. re-identification of ships). From the evaluation on recordings of imagery, it is estimated how well the system discriminates between different ships.
---
paper_title: Recognition of ships for long-term tracking

paper_content:

Long-term tracking is important for maritime situational awareness to identify currently observed ships as earlier encounters. In cases of, for example, piracy and smuggling, past location and behavior analysis are useful to determine whether a ship is of interest. Furthermore, it is beneficial to make this assessment with sensors (such as cameras) at a distance, to avoid costs of bringing an own asset closer to the ship for verification. The emphasis of the research presented in this paper, is on the use of several feature extraction and matching methods for recognizing ships from electro-optical imagery within different categories of vessels. We compared central moments, SIFT with localization and SIFT with Fisher Vectors. From the evaluation on imagery of ships, an indication of discriminative power is obtained between and within different categories of ships. This is used to assess the usefulness in persistent tracking, from short intervals (track improvement) to larger intervals (re-identifying ships). The result of this assessment on real data is used in a simulation environment to determine how track continuity is improved. The simulations showed that even limited recognition will improve tracking, connecting both tracks at short intervals as well as over several days.
---
paper_title: Discriminating small extended targets at sea from clutter and other classes of boats in infrared and visual light imagery

paper_content:

Operating in a coastal environment, with a multitude of boats of different sizes, detection of small extended targets is only one problem. A further difficulty is in discriminating detections of possible threats from alarms due to sea and coastal clutter, and from boats that are neutral for a specific operational task. Adding target features to detections allows filtering out clutter before tracking. Features can also be used to add labels resulting from a classification step. Both will help tracking by facilitating association. Labeling and information from features can be an aid to an operator, or can reduce the number of false alarms for more automatic systems. In this paper we present work on clutter reduction and classification of small extended targets from infrared and visual light imagery. Several methods for discriminating between classes of objects were examined, with an emphasis on less complex techniques, such as rules and decision trees. Similar techniques can be used to discriminate between targets and clutter, and between different classes of boats. Different features are examined that possibly allow discrimination between several classes. Data recordings are used, in infrared and visual light, with a range of targets including rhibs, cabin boats and jet-skis.
---
paper_title: 3-D object modeling from 2-D occluding contour correspondences by opti-acoustic stereo imaging

paper_content:

3-D object modeling from occluding contours from multi-modal optical/sonar stereo.Imaging same contour by zero-baseline stereo, circumventing dense feature matching.Improve reconstruction accuracy from 3-D contour positions and orientations.Identifying degenerate configurations and simple misalignment rectification.Flexibility to use navigation data or a few feature tracks for 3-D trajectory estimation. Utilizing in situ measurements to build 3-D volumetric object models under variety of turbidity conditions is highly desirable for marine sciences. To address the ineffectiveness of feature-based structure from motion and stereo methods under poor visibility, we explore a multi-modal stereo imaging technique that utilizes coincident optical and forward-scan sonar cameras, a so-called opti-acoustic stereo imaging system. The challenges of establishing dense feature correspondences in either opti-acoustic or low-contrast optical stereo images are avoided, by employing 2-D occluding contour correspondences, namely, the images of 3-D object occluding rims. Collecting opti-acoustic stereo pairs while circling an object, matching 2-D apparent contours in optical and sonar views to construct the 3-D occluding rim, and computing the stereo rig trajectory by opti-acoustic bundle adjustment, we generate registered samples of 3-D surface in a reference coordinate system. A surface interpolation gives the 3-D object model.In addition to the key advantage of utilizing range measurements from sonar, the proposed paradigm requires no assumption about local surface curvature as traditionally made in 3-D shape reconstruction from occluding contours. The reconstruction accuracy is improved by computing both the 3-D positions and local surface normals of sampled contours. We also present (1) a simple calibration method to estimate and correct for small discrepancy from the desired relative stereo pose; (2) an analytical analysis of the degenerate configuration that enables special treatment in mapping (tall) elongated objects with dominantly vertical edges. We demonstrate the performance of our method based on the 3-D surface rendering of certain objects, imaged by an underwater opti-acoustic stereo system.
---
paper_title: EO system concepts in the littoral

paper_content:

In recent years, operations executed by naval forces have taken place at many different locations. At present, operations against international terrorism and asymmetric warfare in coastal environments are of major concern. In these scenarios, the threat caused by pirates on-board of small surface targets, such as jetskis and fast inshore attack crafts, is increasing. In the littoral environment, the understanding of its complexity and the efficient use of the limited reaction time, are essential for successful operations. Present-day electro-optical sensor suites, also incorporating Infrared Search and Track systems, can be used for varying tasks as detection, classification and identification. By means of passive electrooptical systems, infrared and visible light sensors, improved situational awareness can be achieved. For long range capability, elevated sensor masts and flying platforms are ideally suited for the surveillance task and improve situational awareness. A primary issue is how to incorporate new electro-optical technology and signal processing into the new sensor concepts, to improve system performance. It is essential to derive accurate information from the high spatial resolution imagery created by the EO sensors. As electro-optical sensors do not have all-weather capability, the performance degradation in adverse scenarios must be understood, in order to support the operational use of adaptive sensor management techniques. In this paper we discuss the approach taken at TNO in the design and assessment of system concepts for future IRST development. An overview of our maritime programme in future IRST and EO system concepts including signal processing is presented.
---
paper_title: Multispectral Target Detection and Tracking for Seaport Video Surveillance

paper_content:

In this paper, a video surveillance process is presented including target detection and tracking of ships at the entrance of a seaport in order to improve security and to prevent terrorist attacks. This process is helpful in the automatic analysis of movements inside the seaport. Steps of detection and tracking are completed using IR data whereas the pattern recognition stage is achieved on color data. A comparison of results of detection and tracking is presented on both IR and color data in order to justify the choice of IR images for these two steps. A draft description of the pattern recognition stage is finally drawn up as development prospect.
---
paper_title: The Autonomous Maritime Navigation (AMN) project: Field tests, autonomous and cooperative behaviors, data fusion, sensors, and vehicles

paper_content:

Many small boat operations can be considered in the categories of “dull, dirty, or dangerous” jobs that are appropriate for automation. Dull jobs include long-range missions or surveillance tasks that can cause physical or mental fatigue in the crew. In addition, operational considerations may limit a human operator's time, such as exposure to heat or cold or union rules on maximum hours per day or job. The marine environment can be a dirty one as well, with wind-driven salt spray damaging unprotected eyes. And military operations such as riverine patrol or interception of potentially hostile craft can be dangerous for a human crew. Robotic systems allowing autonomous small boat operations can be a good match by providing rugged systems that keep sailors out of harm's way and also display “digital acuity” in that their sensors will be as efficient in their first hour of patrolling as in their last. This paper describes work done to build and test an autonomy system allowing several different boats to perform significant missions both by themselves and in cooperative modes. © 2010 Wiley Periodicals, Inc.
---
paper_title: Spatio-temporal alignment of sequences

paper_content:

This paper studies the problem of sequence-to-sequence alignment, namely, establishing correspondences in time and in space between two different video sequences of the same dynamic scene. The sequences are recorded by uncalibrated video cameras which are either stationary or jointly moving, with fixed (but unknown) internal parameters and relative intercamera external parameters. Temporal variations between image frames (such as moving objects or changes in scene illumination) are powerful cues for alignment, which cannot be exploited by standard image-to-image alignment techniques. We show that, by folding spatial and temporal cues into a single alignment framework, situations which are inherently ambiguous for traditional image-to-image alignment methods, are often uniquely resolved by sequence-to-sequence alignment. Furthermore, the ability to align and integrate information across multiple video sequences both in time and in space gives rise to new video applications that are not possible when only image-to-image alignment is used.
---
paper_title: Persistent maritime surveillance using multi-sensor feature association and classification

paper_content:

In maritime operational scenarios, such as smuggling, piracy, or terrorist threats, it is not only relevant who or what an observed object is, but also where it is now and in the past in relation to other (geographical) objects. In situation and impact assessment, this information is used to determine whether an object is a threat. Single platform (ship, harbor) or single sensor information will not provide all this information. The work presented in this paper focuses on the sensor and object levels that provide a description of currently observed objects to situation assessment. For use of information of objects at higher information levels, it is necessary to have not only a good description of observed objects at this moment, but also from its past. Therefore, currently observed objects have to be linked to previous occurrences. Kinematic features, as used in tracking, are of limited use, as uncertainties over longer time intervals are so large that no unique associations can be made. Features extracted from different sensors (e.g., ESM, EO/IR) can be used for both association and classification. Features and classifications are used to associate current objects to previous object descriptions, allowing objects to be described better, and provide position history. In this paper a description of a high level architecture in which such a multi-sensor association is used is described. Results of an assessment of the usability of several features from ESM (from spectrum), EO and IR (shape, contour, keypoints) data for association and classification are shown. © 2012 Copyright Society of Photo-Optical Instrumentation Engineers (SPIE).
---
paper_title: A visual sensing platform for creating a smarter multi-modal marine monitoring network

paper_content:

Demands from various scientific and management communities along with legislative requirements at national and international levels have led to a need for innovative research into large-scale, low-cost, reliable monitoring of our marine and freshwater environments. In this paper we demonstrate the benefits of a multi-modal approach to monitoring and how an in-situ sensor network can be enhanced with the use of contextual image data. We provide an outline of the deployment of a visual sensing system at a busy port and the need for monitoring shipping traffic at the port. Subsequently we present an approach for detecting ships in a challenging image dataset and discuss how this can help to create an intelligent marine monitoring network.
---
paper_title: AUTOMATIC MARITIME SURVEILLANCE WITH VISUAL TARGET DETECTION

paper_content:

In this paper an automatic maritime surveillance system is presented. Boat detection is performed by means of an Haar-like classifier in order to obtain robustness with respect to targets having very different size, reflections and wakes on the water surface, and apparently motionless boats anchored off the coast. Detection results are filtered over the time in order to reduce the false alarm rate. Experimental results show the effectiveness of the approach with different light conditions and camera positions. The system is able to provide the user a global view adding a visual dimension to AIS data.
---
paper_title: Integrated visual information for maritime surveillance

paper_content:

The main contribution of this chapter is to provide a data fusion (DF) scheme for combining in a unique view, radar and visual data. The experimental evaluation of the performance for the modules included in the framework has been carried out using publicly available data from the VOC dataset and the MarDT - Maritime Detection and Tracking (MarDT) data set, containing data coming from different real VTS systems, with ground truth information. Moreover, an operative scenario where traditional VTS systems can benefit from the proposed approach is presented.
---
paper_title: Multisensory data exploitation using advanced image fusion and adaptive colorization

paper_content:

Multisensory data usually present complimentary information such as visual-band imagery and infrared imagery. There ::: is strong evidence that the fused multisensor imagery increases the reliability of interpretation, and the colorized ::: multisensor imagery improves observer performance and reaction times. In this paper, we propose an optimized joint ::: approach of image fusion and colorization in order to synthesize and enhance multisensor imagery such that the resulting ::: imagery can be automatically analyzed by computers (for target recognition) and easily interpreted by human users (for ::: visual analysis). The proposed joint approach provides two sets of synthesized images, a fused image in grayscale and a ::: colorized image in color using a fusion procedure and a colorization procedure, respectively. The proposed image fusion ::: procedure is based on the advanced discrete wavelet (aDWT) transform. The fused image quality (IQ) can be further ::: optimized with respect to an IQ metric by implementing an iterative aDWT procedure. On the other hand, the daylight ::: coloring technique renders the multisensor imagery with natural colors, which human users are use to observing in ::: everyday life. We hereby propose to locally colorize the multisensor imagery segment by mapping the color statistics of ::: the multisensor imagery to that of the daylight images, with which the colorized images resemble daylight pictures. This ::: local coloring procedure also involves histogram analysis, image segmentation, and pattern recognition. The joint fusion ::: and colorization approach can be performed automatically and adaptively regardless of the image contents. Experimental ::: results with multisensor imagery showed that the fused image is informative and clear, and the colored image appears ::: realistic and natural. We anticipate that this optimized joint approach for multisensor imagery will help improve target ::: recognition and visual analysis.
---

Title: Video Processing From Electro-Optical Sensors for Object Detection and Tracking in a Maritime Environment: A Survey

Section 1: INTRODUCTION
Description 1: Write about the importance of maritime surveillance, constraints of traditional methods, and the potential of EO sensors in supporting maritime navigation.

Section 2: MARITIME DATASET FOR COMPARATIVE EVALUATION
Description 2: Discuss the scarcity of publicly available maritime datasets, and introduce the Singapore Maritime Dataset and its features.

Section 3: OBJECT DETECTION
Description 3: Cover the steps involved in object detection, including horizon detection, static background subtraction, and foreground segmentation, along with comparisons of different methods.

Section 4: OBJECT TRACKING
Description 4: Explain how object tracking in maritime environments extends object detection by incorporating temporal data and dynamic background subtraction for more robust modeling.

Section 5: COMPUTER VISION APPROACHES BEYOND MARITIME
Description 5: Explore advanced computer vision techniques from other domains that could potentially be adapted for maritime object detection and tracking.

Section 6: CONCLUDING REMARKS
Description 6: Summarize the findings of the survey, reflect on the current state of maritime EO data processing, and mention future directions for research.

Section 7: Postprocessing of Maritime EO Object Tracking Results
Description 7: Detail methods of post-processing to refine and interpret tracking results in useful physical units and vessel classification.

Section 8: Multisensor Approaches
Description 8: Review how EO sensor data can be effectively combined with other sensor data types like radar and sonar to enhance object detection and tracking.

Section 9: Commercial Maritime Systems
Description 9: Discuss existing commercial maritime systems that integrate EO sensors for applications such as anti-collision and vessel identification.