Understanding vehicular traffic behavior from video: a survey of unsupervised approaches
---
paper_title: Video processing techniques for traffic flow monitoring: A survey

paper_content:

Video-based traffic flow monitoring is a fast emerging field based on the continuous development of computer vision. A survey of the state-of-the-art video processing techniques in traffic flow monitoring is presented in this paper. Firstly, vehicle detection is the first step of video processing and detection methods are classified into background modeling based methods and non-background modeling based methods. In particular, nighttime detection is more challenging due to bad illumination and sensitivity to light. Then tracking techniques, including 3D model-based, region-based, active contour-based and feature-based tracking, are presented. A variety of algorithms including MeanShift algorithm, Kalman Filter and Particle Filter are applied in tracking process. In addition, shadow detection and vehicles occlusion bring much trouble into vehicle detection, tracking and so on. Based on the aforementioned video processing techniques, discussion on behavior understanding including traffic incident detection is carried out. Finally, key challenges in traffic flow monitoring are discussed.
---
paper_title: A Review of Computer Vision Techniques for the Analysis of Urban Traffic

paper_content:

Automatic video analysis from urban surveillance cameras is a fast-emerging field based on computer vision techniques. We present here a comprehensive review of the state-of-the-art computer vision for traffic video with a critical analysis and an outlook to future research directions. This field is of increasing relevance for intelligent transport systems (ITSs). The decreasing hardware cost and, therefore, the increasing deployment of cameras have opened a wide application field for video analytics. Several monitoring objectives such as congestion, traffic rule violation, and vehicle interaction can be targeted using cameras that were typically originally installed for human operators. Systems for the detection and classification of vehicles on highways have successfully been using classical visual surveillance techniques such as background estimation and motion tracking for some time. The urban domain is more challenging with respect to traffic density, lower camera angles that lead to a high degree of occlusion, and the variety of road users. Methods from object categorization and 3-D modeling have inspired more advanced techniques to tackle these challenges. There is no commonly used data set or benchmark challenge, which makes the direct comparison of the proposed algorithms difficult. In addition, evaluation under challenging weather conditions (e.g., rain, fog, and darkness) would be desirable but is rarely performed. Future work should be directed toward robust combined detectors and classifiers for all road users, with a focus on realistic conditions during evaluation.
---
paper_title: Video processing techniques for traffic flow monitoring: A survey

paper_content:

Video-based traffic flow monitoring is a fast emerging field based on the continuous development of computer vision. A survey of the state-of-the-art video processing techniques in traffic flow monitoring is presented in this paper. Firstly, vehicle detection is the first step of video processing and detection methods are classified into background modeling based methods and non-background modeling based methods. In particular, nighttime detection is more challenging due to bad illumination and sensitivity to light. Then tracking techniques, including 3D model-based, region-based, active contour-based and feature-based tracking, are presented. A variety of algorithms including MeanShift algorithm, Kalman Filter and Particle Filter are applied in tracking process. In addition, shadow detection and vehicles occlusion bring much trouble into vehicle detection, tracking and so on. Based on the aforementioned video processing techniques, discussion on behavior understanding including traffic incident detection is carried out. Finally, key challenges in traffic flow monitoring are discussed.
---
paper_title: Trajectory Learning for Activity Understanding: Unsupervised, Multilevel, and Long-Term Adaptive Approach

paper_content:

Society is rapidly accepting the use of video cameras in many new and varied locations, but effective methods to utilize and manage the massive resulting amounts of visual data are only slowly developing. This paper presents a framework for live video analysis in which the behaviors of surveillance subjects are described using a vocabulary learned from recurrent motion patterns, for real-time characterization and prediction of future activities, as well as the detection of abnormalities. The repetitive nature of object trajectories is utilized to automatically build activity models in a 3-stage hierarchical learning process. Interesting nodes are learned through Gaussian mixture modeling, connecting routes formed through trajectory clustering, and spatio-temporal dynamics of activities probabilistically encoded using hidden Markov models. Activity models are adapted to small temporal variations in an online fashion using maximum likelihood regression and new behaviors are discovered from a periodic retraining for long-term monitoring. Extensive evaluation on various data sets, typically missing from other work, demonstrates the efficacy and generality of the proposed framework for surveillance-based activity analysis.
---
paper_title: A Survey of Vision-Based Trajectory Learning and Analysis for Surveillance

paper_content:

This paper presents a survey of trajectory-based activity analysis for visual surveillance. It describes techniques that use trajectory data to define a general set of activities that are applicable to a wide range of scenes and environments. Events of interest are detected by building a generic topographical scene description from underlying motion structure as observed over time. The scene topology is automatically learned and is distinguished by points of interest and motion characterized by activity paths. The methods we review are intended for real-time surveillance through definition of a diverse set of events for further analysis triggering, including virtual fencing, speed profiling, behavior classification, anomaly detection, and object interaction.
---
paper_title: Semantic-Based Surveillance Video Retrieval

paper_content:

Visual surveillance produces large amounts of video data. Effective indexing and retrieval from surveillance video databases are very important. Although there are many ways to represent the content of video clips in current video retrieval algorithms, there still exists a semantic gap between users and retrieval systems. Visual surveillance systems supply a platform for investigating semantic-based video retrieval. In this paper, a semantic-based video retrieval framework for visual surveillance is proposed. A cluster-based tracking algorithm is developed to acquire motion trajectories. The trajectories are then clustered hierarchically using the spatial and temporal information, to learn activity models. A hierarchical structure of semantic indexing and retrieval of object activities, where each individual activity automatically inherits all the semantic descriptions of the activity model to which it belongs, is proposed for accessing video clips and individual objects at the semantic level. The proposed retrieval framework supports various queries including queries by keywords, multiple object queries, and queries by sketch. For multiple object queries, succession and simultaneity restrictions, together with depth and breadth first orders, are considered. For sketch-based queries, a method for matching trajectories drawn by users to spatial trajectories is proposed. The effectiveness and efficiency of our framework are tested in a crowded traffic scene
---
paper_title: Learning Traffic Patterns at Intersections by Spectral Clustering of Motion Trajectories

paper_content:

We address the problem of automatically learning the layout of a traffic intersection from trajectories of vehicles obtained by a vision tracking system. We present a similarity measure which is suitable for use with spectral clustering in problems that emphasize spatial distinctions between vehicle trajectories. The robustness of the method to small perturbations and its sensitivity to the choice of parameters are evaluated using real-world data.
---
paper_title: Fundamentals Of Speech Recognition

paper_content:

1. Fundamentals of Speech Recognition. 2. The Speech Signal: Production, Perception, and Acoustic-Phonetic Characterization. 3. Signal Processing and Analysis Methods for Speech Recognition. 4. Pattern Comparison Techniques. 5. Speech Recognition System Design and Implementation Issues. 6. Theory and Implementation of Hidden Markov Models. 7. Speech Recognition Based on Connected Word Models. 8. Large Vocabulary Continuous Speech Recognition. 9. Task-Oriented Applications of Automatic Speech Recognition.
---
paper_title: Learning semantic scene models from observing activity in visual surveillance

paper_content:

This paper considers the problem of automatically learning an activity-based semantic scene model from a stream of video data. A scene model is proposed that labels regions according to an identifiable activity in each region, such as entry/exit zones, junctions, paths, and stop zones. We present several unsupervised methods that learn these scene elements and present results that show the efficiency of our approach. Finally, we describe how the models can be used to support the interpretation of moving objects in a visual surveillance environment.
---
paper_title: On-line trajectory clustering for anomalous events detection

paper_content:

In this paper, we propose a trajectory clustering algorithm suited for video surveillance systems. Trajectories are clustered on-line, as the data are collected, and clusters are organized in a tree-like structure that, augmented with probability information, can be used to perform behaviour analysis, since it allows the identification of anomalous events.
---
paper_title: Trajectory Learning for Activity Understanding: Unsupervised, Multilevel, and Long-Term Adaptive Approach

paper_content:

Society is rapidly accepting the use of video cameras in many new and varied locations, but effective methods to utilize and manage the massive resulting amounts of visual data are only slowly developing. This paper presents a framework for live video analysis in which the behaviors of surveillance subjects are described using a vocabulary learned from recurrent motion patterns, for real-time characterization and prediction of future activities, as well as the detection of abnormalities. The repetitive nature of object trajectories is utilized to automatically build activity models in a 3-stage hierarchical learning process. Interesting nodes are learned through Gaussian mixture modeling, connecting routes formed through trajectory clustering, and spatio-temporal dynamics of activities probabilistically encoded using hidden Markov models. Activity models are adapted to small temporal variations in an online fashion using maximum likelihood regression and new behaviors are discovered from a periodic retraining for long-term monitoring. Extensive evaluation on various data sets, typically missing from other work, demonstrates the efficacy and generality of the proposed framework for surveillance-based activity analysis.
---
paper_title: Multi feature path modeling for video surveillance

paper_content:

This paper proposes a novel method for detecting nonconforming trajectories of objects as they pass through a scene. Existing methods mostly use spatial features to solve this problem. Using only spatial information is not adequate; we need to take into consideration velocity and curvature information of a trajectory along with the spatial information for an elegant solution. Our method has the ability to distinguish between objects traversing spatially dissimilar paths, or objects traversing spatially proximal paths but having different spatio-temporal characteristics. The method consists of a path building training phase and a testing phase. During the training phase, we use graph-cuts for clustering the trajectories, where the Hausdorff distance metric is used to calculate the edge weights. Each cluster represents a path. An envelope boundary and an average trajectory are computed for each path. During the testing phase we use three features for trajectory matching in a hierarchical fashion. The first feature measures the spatial similarity while the second feature compares the velocity characteristics of trajectories. Finally, the curvature features capture discontinuities in velocity, acceleration, and position of the trajectory. We use real-world pedestrian sequences to demonstrate the practicality of our method.
---
paper_title: Learning common behaviors from large sets of unlabeled temporal series

paper_content:

This paper is about extracting knowledge from large sets of videos, with a particular reference to the video-surveillance application domain. We consider an unsupervised framework and address the specific problem of modeling common behaviors from long-term collection of instantaneous observations. Specifically, such data describe dynamic events and may be represented as time series in an appropriate space of features. Starting off from a set of data meaningful of the common events in a given scenario, the pipeline we propose includes a data abstraction level, that allows us to process different data in a homogeneous way, and a behavior modeling level, based on spectral clustering. At the end of the pipeline we obtain a model of the behaviors which are more frequent in the observed scene, represented by a prototypical behavior, which we call a cluster candidate. We report a detailed experimental evaluation referring to both benchmark datasets and on a complex set of data collected in-house. The experiments show that our method compares very favorably with other approaches from the recent literature. In particular the results we obtain prove that our method is able to capture meaningful information and discard noisy one from very heterogeneous datasets with different levels of prior information available.
---
paper_title: Trajectory Learning for Activity Understanding: Unsupervised, Multilevel, and Long-Term Adaptive Approach

paper_content:

Society is rapidly accepting the use of video cameras in many new and varied locations, but effective methods to utilize and manage the massive resulting amounts of visual data are only slowly developing. This paper presents a framework for live video analysis in which the behaviors of surveillance subjects are described using a vocabulary learned from recurrent motion patterns, for real-time characterization and prediction of future activities, as well as the detection of abnormalities. The repetitive nature of object trajectories is utilized to automatically build activity models in a 3-stage hierarchical learning process. Interesting nodes are learned through Gaussian mixture modeling, connecting routes formed through trajectory clustering, and spatio-temporal dynamics of activities probabilistically encoded using hidden Markov models. Activity models are adapted to small temporal variations in an online fashion using maximum likelihood regression and new behaviors are discovered from a periodic retraining for long-term monitoring. Extensive evaluation on various data sets, typically missing from other work, demonstrates the efficacy and generality of the proposed framework for surveillance-based activity analysis.
---
paper_title: Learning Semantic Motion Patterns for Dynamic Scenes by Improved Sparse Topical Coding

paper_content:

With the proliferation of cameras in public areas, it becomes increasingly desirable to develop fully automated surveillance and monitoring systems. In this paper, we propose a novel unsupervised approach to automatically explore motion patterns occurring in dynamic scenes under an improved sparse topical coding (STC) framework. Given an input video with a fixed camera, we first segment the whole video into a sequence of clips (documents) without overlapping. Optical flow features are extracted from each pair of consecutive frames, and quantized into discrete visual words. Then the video is represented by a word-document hierarchical topic model through a generative process. Finally, an improved sparse topical coding approach is proposed for model learning. The semantic motion patterns (latent topics) are learned automatically and each video clip is represented as a weighted summation of these patterns with only a few nonzero coefficients. The proposed approach is purely data-driven and scene independent (not an object-class specific), which make it suitable for very large range of scenarios. Experiments demonstrate that our approach outperforms the state-of-the art technologies in dynamic scene analysis.
---
paper_title: Large-scale statistical modeling of motion patterns: a Bayesian nonparametric approach

paper_content:

We propose a novel framework for large-scale scene understanding in static camera surveillance. Our techniques combine fast rank-1 constrained robust PCA to compute the foreground, with non-parametric Bayesian models for inference. Clusters are extracted in foreground patterns using a joint multinomial+Gaussian Dirichlet process model (DPM). Since the multinomial distribution is normalized, the Gaussian mixture distinguishes between similar spatial patterns but different activity levels (eg. car vs bike). We propose a modification of the decayed MCMC technique for incremental inference, providing the ability to discover theoretically unlimited patterns in unbounded video streams. A promising by-product of our framework is online, abnormal activity detection. A benchmark video and two surveillance videos, with the longest being 140 hours long are used in our experiments. The patterns discovered are as informative as existing scene understanding algorithms. However, unlike existing work, we achieve near real-time execution and encouraging performance in abnormal activity detection.
---
paper_title: Trajectory Learning for Activity Understanding: Unsupervised, Multilevel, and Long-Term Adaptive Approach

paper_content:

Society is rapidly accepting the use of video cameras in many new and varied locations, but effective methods to utilize and manage the massive resulting amounts of visual data are only slowly developing. This paper presents a framework for live video analysis in which the behaviors of surveillance subjects are described using a vocabulary learned from recurrent motion patterns, for real-time characterization and prediction of future activities, as well as the detection of abnormalities. The repetitive nature of object trajectories is utilized to automatically build activity models in a 3-stage hierarchical learning process. Interesting nodes are learned through Gaussian mixture modeling, connecting routes formed through trajectory clustering, and spatio-temporal dynamics of activities probabilistically encoded using hidden Markov models. Activity models are adapted to small temporal variations in an online fashion using maximum likelihood regression and new behaviors are discovered from a periodic retraining for long-term monitoring. Extensive evaluation on various data sets, typically missing from other work, demonstrates the efficacy and generality of the proposed framework for surveillance-based activity analysis.
---
paper_title: Spectral grouping using the Nystrom method

paper_content:

Spectral graph theoretic methods have recently shown great promise for the problem of image segmentation. However, due to the computational demands of these approaches, applications to large problems such as spatiotemporal data and high resolution imagery have been slow to appear. The contribution of this paper is a method that substantially reduces the computational requirements of grouping algorithms based on spectral partitioning making it feasible to apply them to very large grouping problems. Our approach is based on a technique for the numerical solution of eigenfunction problems known as the Nystrom method. This method allows one to extrapolate the complete grouping solution using only a small number of samples. In doing so, we leverage the fact that there are far fewer coherent groups in a scene than pixels.
---
paper_title: Latent Dirichlet Allocation

paper_content:

We describe latent Dirichlet allocation (LDA), a generative probabilistic model for collections of discrete data such as text corpora. LDA is a three-level hierarchical Bayesian model, in which each item of a collection is modeled as a finite mixture over an underlying set of topics. Each topic is, in turn, modeled as an infinite mixture over an underlying set of topic probabilities. In the context of text modeling, the topic probabilities provide an explicit representation of a document. We present efficient approximate inference techniques based on variational methods and an EM algorithm for empirical Bayes parameter estimation. We report results in document modeling, text classification, and collaborative filtering, comparing to a mixture of unigrams model and the probabilistic LSI model.
---
paper_title: Probabilistic Latent Semantic Analysis

paper_content:

Probabilistic Latent Semantic Analysis is a novel statistical technique for the analysis of two-mode and co-occurrence data, which has applications in information retrieval and filtering, natural language processing, machine learning from text, and in related areas. Compared to standard Latent Semantic Analysis which stems from linear algebra and performs a Singular Value Decomposition of co-occurrence tables, the proposed method is based on a mixture decomposition derived from a latent class model. This results in a more principled approach which has a solid foundation in statistics. In order to avoid overfitting, we propose a widely applicable generalization of maximum likelihood model fitting by tempered EM. Our approach yields substantial and consistent improvements over Latent Semantic Analysis in a number of experiments.
---
paper_title: Learning Semantic Motion Patterns for Dynamic Scenes by Improved Sparse Topical Coding

paper_content:

With the proliferation of cameras in public areas, it becomes increasingly desirable to develop fully automated surveillance and monitoring systems. In this paper, we propose a novel unsupervised approach to automatically explore motion patterns occurring in dynamic scenes under an improved sparse topical coding (STC) framework. Given an input video with a fixed camera, we first segment the whole video into a sequence of clips (documents) without overlapping. Optical flow features are extracted from each pair of consecutive frames, and quantized into discrete visual words. Then the video is represented by a word-document hierarchical topic model through a generative process. Finally, an improved sparse topical coding approach is proposed for model learning. The semantic motion patterns (latent topics) are learned automatically and each video clip is represented as a weighted summation of these patterns with only a few nonzero coefficients. The proposed approach is purely data-driven and scene independent (not an object-class specific), which make it suitable for very large range of scenarios. Experiments demonstrate that our approach outperforms the state-of-the art technologies in dynamic scene analysis.
---
paper_title: An Incremental DPMM-Based Method for Trajectory Clustering, Modeling, and Retrieval

paper_content:

Trajectory analysis is the basis for many applications, such as indexing of motion events in videos, activity recognition, and surveillance. In this paper, the Dirichlet process mixture model (DPMM) is applied to trajectory clustering, modeling, and retrieval. We propose an incremental version of a DPMM-based clustering algorithm and apply it to cluster trajectories. An appropriate number of trajectory clusters is determined automatically. When trajectories belonging to new clusters arrive, the new clusters can be identified online and added to the model without any retraining using the previous data. A time-sensitive Dirichlet process mixture model (tDPMM) is applied to each trajectory cluster for learning the trajectory pattern which represents the time-series characteristics of the trajectories in the cluster. Then, a parameterized index is constructed for each cluster. A novel likelihood estimation algorithm for the tDPMM is proposed, and a trajectory-based video retrieval model is developed. The tDPMM-based probabilistic matching method and the DPMM-based model growing method are combined to make the retrieval model scalable and adaptable. Experimental comparisons with state-of-the-art algorithms demonstrate the effectiveness of our algorithm.
---
paper_title: Unsupervised Activity Perception in Crowded and Complicated Scenes Using Hierarchical Bayesian Models

paper_content:

We propose a novel unsupervised learning framework to model activities and interactions in crowded and complicated scenes. Hierarchical Bayesian models are used to connect three elements in visual surveillance: low-level visual features, simple "atomic" activities, and interactions. Atomic activities are modeled as distributions over low-level visual features, and multi-agent interactions are modeled as distributions over atomic activities. These models are learnt in an unsupervised way. Given a long video sequence, moving pixels are clustered into different atomic activities and short video clips are clustered into different interactions. In this paper, we propose three hierarchical Bayesian models, Latent Dirichlet Allocation (LDA) mixture model, Hierarchical Dirichlet Process (HDP) mixture model, and Dual Hierarchical Dirichlet Processes (Dual-HDP) model. They advance existing language models, such as LDA [1] and HDP [2]. Our data sets are challenging video sequences from crowded traffic scenes and train station scenes with many kinds of activities co-occurring. Without tracking and human labeling effort, our framework completes many challenging visual surveillance tasks of board interest such as: (1) discovering typical atomic activities and interactions; (2) segmenting long video sequences into different interactions; (3) segmenting motions into different activities; (4) detecting abnormality; and (5) supporting high-level queries on activities and interactions.
---
paper_title: Large-scale statistical modeling of motion patterns: a Bayesian nonparametric approach

paper_content:

We propose a novel framework for large-scale scene understanding in static camera surveillance. Our techniques combine fast rank-1 constrained robust PCA to compute the foreground, with non-parametric Bayesian models for inference. Clusters are extracted in foreground patterns using a joint multinomial+Gaussian Dirichlet process model (DPM). Since the multinomial distribution is normalized, the Gaussian mixture distinguishes between similar spatial patterns but different activity levels (eg. car vs bike). We propose a modification of the decayed MCMC technique for incremental inference, providing the ability to discover theoretically unlimited patterns in unbounded video streams. A promising by-product of our framework is online, abnormal activity detection. A benchmark video and two surveillance videos, with the longest being 140 hours long are used in our experiments. The patterns discovered are as informative as existing scene understanding algorithms. However, unlike existing work, we achieve near real-time execution and encouraging performance in abnormal activity detection.
---
paper_title: Detection of dominant flow and abnormal events in surveillance video

paper_content:

We propose an algorithm for abnormal event detection in surveillance video. The proposed algorithm is based on a semi-unsupervised learning method, a kind of feature-based approach so that it does not detect the moving object individually. The proposed algorithm identifies dominant flow without individual object tracking using a latent Dirichlet allocation model in crowded environments. It can also automatically detect and localize an abnormally moving object in real-life video. The performance tests are taken with several real-life databases, and their results show that the proposed algorithm can efficiently detect abnormally moving objects in real time. The proposed algorithm can be applied to any situation in which abnormal directions or abnormal speeds are detected regardless of direction.
---
paper_title: Latent Dirichlet Allocation

paper_content:

We describe latent Dirichlet allocation (LDA), a generative probabilistic model for collections of discrete data such as text corpora. LDA is a three-level hierarchical Bayesian model, in which each item of a collection is modeled as a finite mixture over an underlying set of topics. Each topic is, in turn, modeled as an infinite mixture over an underlying set of topic probabilities. In the context of text modeling, the topic probabilities provide an explicit representation of a document. We present efficient approximate inference techniques based on variational methods and an EM algorithm for empirical Bayes parameter estimation. We report results in document modeling, text classification, and collaborative filtering, comparing to a mixture of unigrams model and the probabilistic LSI model.
---
paper_title: Understanding dynamic scenes by hierarchical motion pattern mining

paper_content:

Our work addresses the problem of analyzing and understanding dynamic video scenes. A two-level motion pattern mining approach is proposed. At the first level, single-agent motion patterns are modeled as distributions over pixel-based features. At the second level, interaction patterns are modeled as distributions over single-agent motion patterns. Both patterns are shared among video clips. Compared to other works, the advantage of our method is that interaction patterns are detected and assigned to every video frame. This enables a finer semantic interpretation and more precise anomaly detection. Specifically, every video frame is labeled by a certain interaction pattern and moving pixels in each frame which do not belong to any singleagent pattern or cannot exist in the corresponding interaction pattern are detected as anomalies. We have tested our approach on a challenging traffic surveillance sequence containing both pedestrian and vehicular motions and obtained promising results.
---
paper_title: Learning Behavioural Context

paper_content:

We propose a novel framework for automatic discovering and learning of behavioural context for video-based complex behaviour recognition and anomaly detection. Our work differs from most previous efforts on learning visual context in that our model learns multi-scale spatio-temporal rather than static context. Specifically three types of behavioural context are investigated: behaviour spatial context, behaviour correlation context, and behaviour temporal context. To that end, the proposed framework consists of an activity-based semantic scene segmentation model for learning behaviour spatial context, and a cascaded probabilistic topic model for learning both behaviour correlation context and behaviour temporal context at multiple scales. These behaviour context models are deployed for recognising non-exaggerated multi-object interactive and co-existence behaviours in public spaces. In particular, we develop a method for detecting subtle behavioural anomalies against the learned context. The effectiveness of the proposed approach is validated by extensive experiments carried out using data captured from complex and crowded outdoor scenes.
---
paper_title: Video Behaviour Mining Using a Dynamic Topic Model

paper_content:

This paper addresses the problem of fully automated mining of public space video data, a highly desirable capability under contemporary commercial and security considerations. This task is especially challenging due to the complexity of the object behaviors to be profiled, the difficulty of analysis under the visual occlusions and ambiguities common in public space video, and the computational challenge of doing so in real-time. We address these issues by introducing a new dynamic topic model, termed a Markov Clustering Topic Model (MCTM). The MCTM builds on existing dynamic Bayesian network models and Bayesian topic models, and overcomes their drawbacks on sensitivity, robustness and efficiency. Specifically, our model profiles complex dynamic scenes by robustly clustering visual events into activities and these activities into global behaviours with temporal dynamics. A Gibbs sampler is derived for offline learning with unlabeled training data and a new approximation to online Bayesian inference is formulated to enable dynamic scene understanding and behaviour mining in new video data online in real-time. The strength of this model is demonstrated by unsupervised learning of dynamic scene models for four complex and crowded public scenes, and successful mining of behaviors and detection of salient events in each.
---
paper_title: A Sequential Topic Model for Mining Recurrent Activities from Long Term Video Logs

paper_content:

This paper introduces a novel probabilistic activity modeling approach that mines recurrent sequential patterns called motifs from documents given as word \(\times \) time count matrices (e.g., videos). In this model, documents are represented as a mixture of sequential activity patterns (our motifs) where the mixing weights are defined by the motif starting time occurrences. The novelties are multi fold. First, unlike previous approaches where topics modeled only the co-occurrence of words at a given time instant, our motifs model the co-occurrence and temporal order in which the words occur within a temporal window. Second, unlike traditional Dynamic Bayesian networks (DBN), our model accounts for the important case where activities occur concurrently in the video (but not necessarily in synchrony), i.e., the advent of activity motifs can overlap. The learning of the motifs in these difficult situations is made possible thanks to the introduction of latent variables representing the activity starting times, enabling us to implicitly align the occurrences of the same pattern during the joint inference of the motifs and their starting times. As a third novelty, we propose a general method that favors the recovery of sparse distributions, a highly desirable property in many topic model applications, by adding simple regularization constraints on the searched distributions to the data likelihood optimization criteria. We substantiate our claims with experiments on synthetic data to demonstrate the algorithm behavior, and on four video datasets with significant variations in their activity content obtained from static cameras. We observe that using low-level motion features from videos, our algorithm is able to capture sequential patterns that implicitly represent typical trajectories of scene objects.
---
paper_title: Bridging the past, present and future: Modeling scene activities from event relationships and global rules

paper_content:

This paper addresses the discovery of activities and learns the underlying processes that govern their occurrences over time in complex surveillance scenes. To this end, we propose a novel topic model that accounts for the two main factors that affect these occurrences: (1) the existence of global scene states that regulate which of the activities can spontaneously occur; (2) local rules that link past activity occurrences to current ones with temporal lags. These complementary factors are mixed in the probabilistic generative process, thanks to the use of a binary random variable that selects for each activity occurrence which one of the above two factors is applicable. All model parameters are efficiently inferred using a collapsed Gibbs sampling inference scheme. Experiments on various datasets from the literature show that the model is able to capture temporal processes at multiple scales: the scene-level first order Markovian process, and causal relationships amongst activities that can be used to predict which activity can happen after another one, and after what delay, thus providing a rich interpretation of the scene's dynamical content.
---
paper_title: What's going on? Discovering spatio-temporal dependencies in dynamic scenes

paper_content:

We present two novel methods to automatically learn spatio-temporal dependencies of moving agents in complex dynamic scenes. They allow to discover temporal rules, such as the right of way between different lanes or typical traffic light sequences. To extract them, sequences of activities need to be learned. While the first method extracts rules based on a learned topic model, the second model called DDP-HMM jointly learns co-occurring activities and their time dependencies. To this end we employ Dependent Dirichlet Processes to learn an arbitrary number of infinite Hidden Markov Models. In contrast to previous work, we build on state-of-the-art topic models that allow to automatically infer all parameters such as the optimal number of HMMs necessary to explain the rules governing a scene. The models are trained offline by Gibbs Sampling using unlabeled training data.
---
paper_title: Action Recognition Using Topic Models

paper_content:

In this book chapter, we will introduce approaches of using topic models for action recognition. Topic models were originally developed in language processing. In recent years, they were applied to action recognition and other computer vision problems, and achieved great success. Topic models are unsupervised. The models of actions are learned through exploring the co-occurrence of visual features without manually labeled training examples. This is important when there are a large number of actions to be recognized in a large variety of scenes. Most topic models are hierarchical Bayesian models and they jointly model simple actions and complicated actions at different hierarchical levels. Knowledge and contextual information can be well integrated into topic models as priors. We will explain how topic models can be used in different ways for action recognition in different scenarios. For examples, the scenes may be sparse or crowded. There may be a single camera view or multiple camera views. The camera settings may be near-field or far-field. In different scenarios, different features, such as trajectories, local motions and spatial-temporal interest points, are used for action recognition.
---
paper_title: An Incremental DPMM-Based Method for Trajectory Clustering, Modeling, and Retrieval

paper_content:

Trajectory analysis is the basis for many applications, such as indexing of motion events in videos, activity recognition, and surveillance. In this paper, the Dirichlet process mixture model (DPMM) is applied to trajectory clustering, modeling, and retrieval. We propose an incremental version of a DPMM-based clustering algorithm and apply it to cluster trajectories. An appropriate number of trajectory clusters is determined automatically. When trajectories belonging to new clusters arrive, the new clusters can be identified online and added to the model without any retraining using the previous data. A time-sensitive Dirichlet process mixture model (tDPMM) is applied to each trajectory cluster for learning the trajectory pattern which represents the time-series characteristics of the trajectories in the cluster. Then, a parameterized index is constructed for each cluster. A novel likelihood estimation algorithm for the tDPMM is proposed, and a trajectory-based video retrieval model is developed. The tDPMM-based probabilistic matching method and the DPMM-based model growing method are combined to make the retrieval model scalable and adaptable. Experimental comparisons with state-of-the-art algorithms demonstrate the effectiveness of our algorithm.
---
paper_title: A Non-parametric Hierarchical Model to Discover Behavior Dynamics from Tracks

paper_content:

We present a novel non-parametric Bayesian model to jointly discover the dynamics of low-level actions and high-level behaviors of tracked people in open environments. Our model represents behaviors as Markov chains of actions which capture high-level temporal dynamics. Actions may be shared by various behaviors and represent spatially localized occurrences of a person's low-level motion dynamics using Switching Linear Dynamics Systems. Since the model handles real-valued features directly, we do not lose information by quantizing measurements to 'visual words' and can thus discover variations in standing, walking and running without discrete thresholds. We describe inference using Gibbs sampling and validate our approach on several artificial and real-world tracking datasets. We show that our model can distinguish relevant behavior patterns that an existing state-of-the-art method for hierarchical clustering cannot.
---
paper_title: Action Recognition Using Topic Models

paper_content:

In this book chapter, we will introduce approaches of using topic models for action recognition. Topic models were originally developed in language processing. In recent years, they were applied to action recognition and other computer vision problems, and achieved great success. Topic models are unsupervised. The models of actions are learned through exploring the co-occurrence of visual features without manually labeled training examples. This is important when there are a large number of actions to be recognized in a large variety of scenes. Most topic models are hierarchical Bayesian models and they jointly model simple actions and complicated actions at different hierarchical levels. Knowledge and contextual information can be well integrated into topic models as priors. We will explain how topic models can be used in different ways for action recognition in different scenarios. For examples, the scenes may be sparse or crowded. There may be a single camera view or multiple camera views. The camera settings may be near-field or far-field. In different scenarios, different features, such as trajectories, local motions and spatial-temporal interest points, are used for action recognition.
---
paper_title: Trajectory Analysis and Semantic Region Modeling Using Nonparametric Hierarchical Bayesian Models

paper_content:

We propose a novel framework of using a nonparametric Bayesian model, called Dual Hierarchical Dirichlet Processes (Dual-HDP) (Wang et al. in IEEE Trans. Pattern Anal. Mach. Intell. 31:539---555, 2009), for unsupervised trajectory analysis and semantic region modeling in surveillance settings. In our approach, trajectories are treated as documents and observations of an object on a trajectory are treated as words in a document. Trajectories are clustered into different activities. Abnormal trajectories are detected as samples with low likelihoods. The semantic regions, which are subsets of paths commonly taken by objects and are related to activities in the scene, are also modeled. Under Dual-HDP, both the number of activity categories and the number of semantic regions are automatically learnt from data. In this paper, we further extend Dual-HDP to a Dynamic Dual-HDP model which allows dynamic update of activity models and online detection of normal/abnormal activities. Experiments are evaluated on a simulated data set and two real data sets, which include 8,478 radar tracks collected from a maritime port and 40,453 visual tracks collected from a parking lot.
---
paper_title: Trajectory Analysis and Semantic Region Modeling Using Nonparametric Hierarchical Bayesian Models

paper_content:

We propose a novel framework of using a nonparametric Bayesian model, called Dual Hierarchical Dirichlet Processes (Dual-HDP) (Wang et al. in IEEE Trans. Pattern Anal. Mach. Intell. 31:539---555, 2009), for unsupervised trajectory analysis and semantic region modeling in surveillance settings. In our approach, trajectories are treated as documents and observations of an object on a trajectory are treated as words in a document. Trajectories are clustered into different activities. Abnormal trajectories are detected as samples with low likelihoods. The semantic regions, which are subsets of paths commonly taken by objects and are related to activities in the scene, are also modeled. Under Dual-HDP, both the number of activity categories and the number of semantic regions are automatically learnt from data. In this paper, we further extend Dual-HDP to a Dynamic Dual-HDP model which allows dynamic update of activity models and online detection of normal/abnormal activities. Experiments are evaluated on a simulated data set and two real data sets, which include 8,478 radar tracks collected from a maritime port and 40,453 visual tracks collected from a parking lot.
---
paper_title: Earth mover's prototypes: A convex learning approach for discovering activity patterns in dynamic scenes

paper_content:

We present a novel approach for automatically discovering spatio-temporal patterns in complex dynamic scenes. Similarly to recent non-object centric methods, we use low level visual cues to detect atomic activities and then construct clip histograms. Differently from previous works, we formulate the task of discovering high level activity patterns as a prototype learning problem where the correlation among atomic activities is explicitly taken into account when grouping clip histograms. Interestingly at the core of our approach there is a convex optimization problem which allows us to efficiently extract patterns at multiple levels of detail. The effectiveness of our method is demonstrated on publicly available datasets.
---
paper_title: Exploiting sparse representations for robust analysis of noisy complex video scenes

paper_content:

Recent works have shown that, even with simple low level visual cues, complex behaviors can be extracted automatically from crowded scenes, e.g. those depicting public spaces recorded from video surveillance cameras. However, low level features as optical flow or foreground pixels are inherently noisy. In this paper we propose a novel unsupervised learning approach for the analysis of complex scenes which is specifically tailored to cope directly with features' noise and uncertainty. We formalize the task of extracting activity patterns as a matrix factorization problem, considering as reconstruction function the robust Earth Mover's Distance. A constraint of sparsity on the computed basis matrix is imposed, filtering out noise and leading to the identification of the most relevant elementary activities in a typical high level behavior. We further derive an alternate optimization approach to solve the proposed problem efficiently and we show that it is reduced to a sequence of linear programs. Finally, we propose to use short trajectory snippets to account for object motion information, in alternative to the noisy optical flow vectors used in previous works. Experimental results demonstrate that our method yields similar or superior performance to state-of-the arts approaches.
---
paper_title: Scene understanding by statistical modeling of motion patterns

paper_content:

We present a novel method for the discovery and statistical representation of motion patterns in a scene observed by a static camera. Related methods involving learning of patterns of activity rely on trajectories obtained from object detection and tracking systems, which are unreliable in complex scenes of crowded motion. We propose a mixture model representation of salient patterns of optical flow, and present an algorithm for learning these patterns from dense optical flow in a hierarchical, unsupervised fashion. Using low level cues of noisy optical flow, K-means is employed to initialize a Gaussian mixture model for temporally segmented clips of video. The components of this mixture are then filtered and instances of motion patterns are computed using a simple motion model, by linking components across space and time. Motion patterns are then initialized and membership of instances in different motion patterns is established by using KL divergence between mixture distributions of pattern instances. Finally, a pixel level representation of motion patterns is proposed by deriving conditional expectation of optical flow. Results of extensive experiments are presented for multiple surveillance sequences containing numerous patterns involving both pedestrian and vehicular traffic.
---
paper_title: Unsupervised Activity Perception in Crowded and Complicated Scenes Using Hierarchical Bayesian Models

paper_content:

We propose a novel unsupervised learning framework to model activities and interactions in crowded and complicated scenes. Hierarchical Bayesian models are used to connect three elements in visual surveillance: low-level visual features, simple "atomic" activities, and interactions. Atomic activities are modeled as distributions over low-level visual features, and multi-agent interactions are modeled as distributions over atomic activities. These models are learnt in an unsupervised way. Given a long video sequence, moving pixels are clustered into different atomic activities and short video clips are clustered into different interactions. In this paper, we propose three hierarchical Bayesian models, Latent Dirichlet Allocation (LDA) mixture model, Hierarchical Dirichlet Process (HDP) mixture model, and Dual Hierarchical Dirichlet Processes (Dual-HDP) model. They advance existing language models, such as LDA [1] and HDP [2]. Our data sets are challenging video sequences from crowded traffic scenes and train station scenes with many kinds of activities co-occurring. Without tracking and human labeling effort, our framework completes many challenging visual surveillance tasks of board interest such as: (1) discovering typical atomic activities and interactions; (2) segmenting long video sequences into different interactions; (3) segmenting motions into different activities; (4) detecting abnormality; and (5) supporting high-level queries on activities and interactions.
---
paper_title: Automatic adaptation of a generic pedestrian detector to a specific traffic scene

paper_content:

In recent years significant progress has been made learning generic pedestrian detectors from manually labeled large scale training sets. However, when a generic pedestrian detector is applied to a specific scene where the testing data does not match with the training data because of variations of viewpoints, resolutions, illuminations and backgrounds, its accuracy may decrease greatly. In this paper, we propose a new framework of adapting a pre-trained generic pedestrian detector to a specific traffic scene by automatically selecting both confident positive and negative examples from the target scene to re-train the detector iteratively. An important feature of the proposed framework is to utilize unsupervisedly learned models of vehicle and pedestrian paths, together with multiple other cues such as locations, sizes, appearance and motions to select new training samples. The information of scene structures increases the reliability of selected samples and is complementary to the appearance-based detector. However, it was not well explored in previous studies. In order to further improve the reliability of selected samples, outliers are removed through multiple hierarchical clustering steps. The effectiveness of different cues and clustering steps is evaluated through experiments. The proposed approach significantly improves the accuracy of the generic pedestrian detector and also outperforms the scene specific detector retrained using background subtraction. Its results are comparable with the detector trained using a large number of manually labeled frames from the target scene.
---
paper_title: Trajectory analysis and semantic region modeling using a nonparametric Bayesian model

paper_content:

We propose a novel nonparametric Bayesian model, Dual Hierarchical Dirichlet Processes (Dual-HDP), for trajectory analysis and semantic region modeling in surveillance settings, in an unsupervised way. In our approach, trajectories are treated as documents and observations of an object on a trajectory are treated as words in a document. Trajectories are clustered into different activities. Abnormal trajectories are detected as samples with low likelihoods. The semantic regions, which are intersections of paths commonly taken by objects, related to activities in the scene are also modeled. Dual-HDP advances the existing Hierarchical Dirichlet Processes (HDP) language model. HDP only clusters co-occurring words from documents into topics and automatically decides the number of topics. Dual-HDP co-clusters both words and documents. It learns both the numbers of word topics and document clusters from data. Under our problem settings, HDP only clusters observations of objects, while Dual-HDP clusters both observations and trajectories. Experiments are evaluated on two data sets, radar tracks collected from a maritime port and visual tracks collected from a parking lot.
---
paper_title: Trajectory Learning for Activity Understanding: Unsupervised, Multilevel, and Long-Term Adaptive Approach

paper_content:

Society is rapidly accepting the use of video cameras in many new and varied locations, but effective methods to utilize and manage the massive resulting amounts of visual data are only slowly developing. This paper presents a framework for live video analysis in which the behaviors of surveillance subjects are described using a vocabulary learned from recurrent motion patterns, for real-time characterization and prediction of future activities, as well as the detection of abnormalities. The repetitive nature of object trajectories is utilized to automatically build activity models in a 3-stage hierarchical learning process. Interesting nodes are learned through Gaussian mixture modeling, connecting routes formed through trajectory clustering, and spatio-temporal dynamics of activities probabilistically encoded using hidden Markov models. Activity models are adapted to small temporal variations in an online fashion using maximum likelihood regression and new behaviors are discovered from a periodic retraining for long-term monitoring. Extensive evaluation on various data sets, typically missing from other work, demonstrates the efficacy and generality of the proposed framework for surveillance-based activity analysis.
---
paper_title: Video Behaviour Mining Using a Dynamic Topic Model

paper_content:

This paper addresses the problem of fully automated mining of public space video data, a highly desirable capability under contemporary commercial and security considerations. This task is especially challenging due to the complexity of the object behaviors to be profiled, the difficulty of analysis under the visual occlusions and ambiguities common in public space video, and the computational challenge of doing so in real-time. We address these issues by introducing a new dynamic topic model, termed a Markov Clustering Topic Model (MCTM). The MCTM builds on existing dynamic Bayesian network models and Bayesian topic models, and overcomes their drawbacks on sensitivity, robustness and efficiency. Specifically, our model profiles complex dynamic scenes by robustly clustering visual events into activities and these activities into global behaviours with temporal dynamics. A Gibbs sampler is derived for offline learning with unlabeled training data and a new approximation to online Bayesian inference is formulated to enable dynamic scene understanding and behaviour mining in new video data online in real-time. The strength of this model is demonstrated by unsupervised learning of dynamic scene models for four complex and crowded public scenes, and successful mining of behaviors and detection of salient events in each.
---
paper_title: Video Behaviour Mining Using a Dynamic Topic Model

paper_content:

This paper addresses the problem of fully automated mining of public space video data, a highly desirable capability under contemporary commercial and security considerations. This task is especially challenging due to the complexity of the object behaviors to be profiled, the difficulty of analysis under the visual occlusions and ambiguities common in public space video, and the computational challenge of doing so in real-time. We address these issues by introducing a new dynamic topic model, termed a Markov Clustering Topic Model (MCTM). The MCTM builds on existing dynamic Bayesian network models and Bayesian topic models, and overcomes their drawbacks on sensitivity, robustness and efficiency. Specifically, our model profiles complex dynamic scenes by robustly clustering visual events into activities and these activities into global behaviours with temporal dynamics. A Gibbs sampler is derived for offline learning with unlabeled training data and a new approximation to online Bayesian inference is formulated to enable dynamic scene understanding and behaviour mining in new video data online in real-time. The strength of this model is demonstrated by unsupervised learning of dynamic scene models for four complex and crowded public scenes, and successful mining of behaviors and detection of salient events in each.
---
paper_title: Unsupervised Activity Perception in Crowded and Complicated Scenes Using Hierarchical Bayesian Models

paper_content:

We propose a novel unsupervised learning framework to model activities and interactions in crowded and complicated scenes. Hierarchical Bayesian models are used to connect three elements in visual surveillance: low-level visual features, simple "atomic" activities, and interactions. Atomic activities are modeled as distributions over low-level visual features, and multi-agent interactions are modeled as distributions over atomic activities. These models are learnt in an unsupervised way. Given a long video sequence, moving pixels are clustered into different atomic activities and short video clips are clustered into different interactions. In this paper, we propose three hierarchical Bayesian models, Latent Dirichlet Allocation (LDA) mixture model, Hierarchical Dirichlet Process (HDP) mixture model, and Dual Hierarchical Dirichlet Processes (Dual-HDP) model. They advance existing language models, such as LDA [1] and HDP [2]. Our data sets are challenging video sequences from crowded traffic scenes and train station scenes with many kinds of activities co-occurring. Without tracking and human labeling effort, our framework completes many challenging visual surveillance tasks of board interest such as: (1) discovering typical atomic activities and interactions; (2) segmenting long video sequences into different interactions; (3) segmenting motions into different activities; (4) detecting abnormality; and (5) supporting high-level queries on activities and interactions.
---
paper_title: Trajectory analysis and semantic region modeling using a nonparametric Bayesian model

paper_content:

We propose a novel nonparametric Bayesian model, Dual Hierarchical Dirichlet Processes (Dual-HDP), for trajectory analysis and semantic region modeling in surveillance settings, in an unsupervised way. In our approach, trajectories are treated as documents and observations of an object on a trajectory are treated as words in a document. Trajectories are clustered into different activities. Abnormal trajectories are detected as samples with low likelihoods. The semantic regions, which are intersections of paths commonly taken by objects, related to activities in the scene are also modeled. Dual-HDP advances the existing Hierarchical Dirichlet Processes (HDP) language model. HDP only clusters co-occurring words from documents into topics and automatically decides the number of topics. Dual-HDP co-clusters both words and documents. It learns both the numbers of word topics and document clusters from data. Under our problem settings, HDP only clusters observations of objects, while Dual-HDP clusters both observations and trajectories. Experiments are evaluated on two data sets, radar tracks collected from a maritime port and visual tracks collected from a parking lot.
---
paper_title: Trajectory Learning for Activity Understanding: Unsupervised, Multilevel, and Long-Term Adaptive Approach

paper_content:

Society is rapidly accepting the use of video cameras in many new and varied locations, but effective methods to utilize and manage the massive resulting amounts of visual data are only slowly developing. This paper presents a framework for live video analysis in which the behaviors of surveillance subjects are described using a vocabulary learned from recurrent motion patterns, for real-time characterization and prediction of future activities, as well as the detection of abnormalities. The repetitive nature of object trajectories is utilized to automatically build activity models in a 3-stage hierarchical learning process. Interesting nodes are learned through Gaussian mixture modeling, connecting routes formed through trajectory clustering, and spatio-temporal dynamics of activities probabilistically encoded using hidden Markov models. Activity models are adapted to small temporal variations in an online fashion using maximum likelihood regression and new behaviors are discovered from a periodic retraining for long-term monitoring. Extensive evaluation on various data sets, typically missing from other work, demonstrates the efficacy and generality of the proposed framework for surveillance-based activity analysis.
---
paper_title: Topic models for scene analysis and abnormality detection

paper_content:

Automatic analysis and understanding of common activities and detection of deviant behaviors is a challenging task in computer vision. This is particularly true in surveillance data, where busy traffic scenes are rich with multifarious activities many of them occurring simultaneously. In this paper, we address these issues with an unsupervised learning approach relying on probabilistic Latent Semantic Analysis (pLSA) applied to a rich set visual features including motion and size activities for discovering relevant activity patterns occurring in such scenes. We then show how the discovered patterns can directly be used to segment the scene into regions with clear semantic activity content. Furthermore, we introduce novel abnormality detection measures within the scope of the adopted modeling approach, and investigate in detail their performance with respect to various issues. Experiments on 45 minutes of video captured from a busy traffic scene and involving abnormal events are conducted.
---
paper_title: Comparison of Similarity Measures for Trajectory Clustering in Outdoor Surveillance Scenes

paper_content:

This paper compares different similarity measures used for trajectory clustering in outdoor surveillance scenes. Six similarity measures are presented and the performance is evaluated by Correct Clustering Rate (CCR) and Time Cost (TC). The experimental results demonstrate that in outdoor surveillance scenes, the simpler PCA+Euclidean distance is competent for the clustering task even in case of noise, as more complex similarity measures such as DTW, LCSS are not efficient due to their high computational cost .
---
paper_title: An Incremental DPMM-Based Method for Trajectory Clustering, Modeling, and Retrieval

paper_content:

Trajectory analysis is the basis for many applications, such as indexing of motion events in videos, activity recognition, and surveillance. In this paper, the Dirichlet process mixture model (DPMM) is applied to trajectory clustering, modeling, and retrieval. We propose an incremental version of a DPMM-based clustering algorithm and apply it to cluster trajectories. An appropriate number of trajectory clusters is determined automatically. When trajectories belonging to new clusters arrive, the new clusters can be identified online and added to the model without any retraining using the previous data. A time-sensitive Dirichlet process mixture model (tDPMM) is applied to each trajectory cluster for learning the trajectory pattern which represents the time-series characteristics of the trajectories in the cluster. Then, a parameterized index is constructed for each cluster. A novel likelihood estimation algorithm for the tDPMM is proposed, and a trajectory-based video retrieval model is developed. The tDPMM-based probabilistic matching method and the DPMM-based model growing method are combined to make the retrieval model scalable and adaptable. Experimental comparisons with state-of-the-art algorithms demonstrate the effectiveness of our algorithm.
---
paper_title: Learning trajectory patterns by clustering: Experimental studies and comparative evaluation

paper_content:

Recently a large amount of research has been devoted to automatic activity analysis. Typically, activities have been defined by their motion characteristics and represented by trajectories. These trajectories are collected and clustered to determine typical behaviors. This paper evaluates different similarity measures and clustering methodologies to catalog their strengths and weaknesses when utilized for the trajectory learning problem. The clustering performance is measured by evaluating the correct clustering rate on different datasets with varying characteristics.
---
paper_title: Trajectory Analysis and Semantic Region Modeling Using Nonparametric Hierarchical Bayesian Models

paper_content:

We propose a novel framework of using a nonparametric Bayesian model, called Dual Hierarchical Dirichlet Processes (Dual-HDP) (Wang et al. in IEEE Trans. Pattern Anal. Mach. Intell. 31:539---555, 2009), for unsupervised trajectory analysis and semantic region modeling in surveillance settings. In our approach, trajectories are treated as documents and observations of an object on a trajectory are treated as words in a document. Trajectories are clustered into different activities. Abnormal trajectories are detected as samples with low likelihoods. The semantic regions, which are subsets of paths commonly taken by objects and are related to activities in the scene, are also modeled. Under Dual-HDP, both the number of activity categories and the number of semantic regions are automatically learnt from data. In this paper, we further extend Dual-HDP to a Dynamic Dual-HDP model which allows dynamic update of activity models and online detection of normal/abnormal activities. Experiments are evaluated on a simulated data set and two real data sets, which include 8,478 radar tracks collected from a maritime port and 40,453 visual tracks collected from a parking lot.
---
paper_title: Evaluation of fiber clustering methods for diffusion tensor imaging

paper_content:

Fiber tracking is a standard approach for the visualization of the results of diffusion tensor imaging (DTI). If fibers are reconstructed and visualized individually through the complete white matter, the display gets easily cluttered making it difficult to get insight in the data. Various clustering techniques have been proposed to automatically obtain bundles that should represent anatomical structures, but it is unclear which clustering methods and parameter settings give the best results. We propose a framework to validate clustering methods for white-matter fibers. Clusters are compared with a manual classification which is used as a ground truth. For the quantitative evaluation of the methods, we developed a new measure to assess the difference between the ground truth and the clusterings. The measure was validated and calibrated by presenting different clusterings to physicians and asking them for their judgement. We found that the values of our new measure for different clusterings match well with the opinions of physicians. Using this framework, we have evaluated different clustering algorithms, including shared nearest neighbor clustering, which has not been used before for this purpose. We found that the use of hierarchical clustering using single-link and a fiber similarity measure based on the mean distance between fibers gave the best results.
---
paper_title: Video Behaviour Mining Using a Dynamic Topic Model

paper_content:

This paper addresses the problem of fully automated mining of public space video data, a highly desirable capability under contemporary commercial and security considerations. This task is especially challenging due to the complexity of the object behaviors to be profiled, the difficulty of analysis under the visual occlusions and ambiguities common in public space video, and the computational challenge of doing so in real-time. We address these issues by introducing a new dynamic topic model, termed a Markov Clustering Topic Model (MCTM). The MCTM builds on existing dynamic Bayesian network models and Bayesian topic models, and overcomes their drawbacks on sensitivity, robustness and efficiency. Specifically, our model profiles complex dynamic scenes by robustly clustering visual events into activities and these activities into global behaviours with temporal dynamics. A Gibbs sampler is derived for offline learning with unlabeled training data and a new approximation to online Bayesian inference is formulated to enable dynamic scene understanding and behaviour mining in new video data online in real-time. The strength of this model is demonstrated by unsupervised learning of dynamic scene models for four complex and crowded public scenes, and successful mining of behaviors and detection of salient events in each.
---
paper_title: Anomalous video event detection using spatiotemporal context

paper_content:

Compared to other anomalous video event detection approaches that analyze object trajectories only, we propose a context-aware method to detect anomalies. By tracking all moving objects in the video, three different levels of spatiotemporal contexts are considered, i.e., point anomaly of a video object, sequential anomaly of an object trajectory, and co-occurrence anomaly of multiple video objects. A hierarchical data mining approach is proposed. At each level, frequency-based analysis is performed to automatically discover regular rules of normal events. Events deviating from these rules are identified as anomalies. The proposed method is computationally efficient and can infer complex rules. Experiments on real traffic video validate that the detected video anomalies are hazardous or illegal according to traffic regulations.
---
paper_title: Trajectory Learning for Activity Understanding: Unsupervised, Multilevel, and Long-Term Adaptive Approach

paper_content:

Society is rapidly accepting the use of video cameras in many new and varied locations, but effective methods to utilize and manage the massive resulting amounts of visual data are only slowly developing. This paper presents a framework for live video analysis in which the behaviors of surveillance subjects are described using a vocabulary learned from recurrent motion patterns, for real-time characterization and prediction of future activities, as well as the detection of abnormalities. The repetitive nature of object trajectories is utilized to automatically build activity models in a 3-stage hierarchical learning process. Interesting nodes are learned through Gaussian mixture modeling, connecting routes formed through trajectory clustering, and spatio-temporal dynamics of activities probabilistically encoded using hidden Markov models. Activity models are adapted to small temporal variations in an online fashion using maximum likelihood regression and new behaviors are discovered from a periodic retraining for long-term monitoring. Extensive evaluation on various data sets, typically missing from other work, demonstrates the efficacy and generality of the proposed framework for surveillance-based activity analysis.
---
paper_title: Learning Behavioural Context

paper_content:

We propose a novel framework for automatic discovering and learning of behavioural context for video-based complex behaviour recognition and anomaly detection. Our work differs from most previous efforts on learning visual context in that our model learns multi-scale spatio-temporal rather than static context. Specifically three types of behavioural context are investigated: behaviour spatial context, behaviour correlation context, and behaviour temporal context. To that end, the proposed framework consists of an activity-based semantic scene segmentation model for learning behaviour spatial context, and a cascaded probabilistic topic model for learning both behaviour correlation context and behaviour temporal context at multiple scales. These behaviour context models are deployed for recognising non-exaggerated multi-object interactive and co-existence behaviours in public spaces. In particular, we develop a method for detecting subtle behavioural anomalies against the learned context. The effectiveness of the proposed approach is validated by extensive experiments carried out using data captured from complex and crowded outdoor scenes.
---
paper_title: Delta-Dual Hierarchical Dirichlet Processes: A pragmatic abnormal behaviour detector

paper_content:

In the security domain a key problem is identifying rare behaviours of interest. Training examples for these behaviours may or may not exist, and if they do exist there will be few examples, quite probably one. We present a novel weakly supervised algorithm that can detect behaviours that either have never before been seen or for which there are few examples. Global context is modelled, allowing the detection of abnormal behaviours that in isolation appear normal. Pragmatic aspects are considered, such that no parameter tuning is required and real time performance is achieved.
---
paper_title: Real-Time Video-Based Traffic Measurement and Visualization System for Energy/Emissions

paper_content:

The ability to monitor the state of a given roadway in order to better manage traffic congestion has become increasingly important. Sophisticated traffic management systems able to process both the static and mobile sensor data and provide traffic information for the roadway network are under development. In addition to typical traffic data such as flow, density, and average traffic speed, there is now strong interest in environmental factors such as greenhouse gases, pollutant emissions, and fuel consumption. It is now possible to combine high-resolution real-time traffic data with instantaneous emission models to estimate these environmental measures in real time. In this paper, a system that estimates average traffic fuel economy, CO2 , CO, HC, and NOx emissions using a computer-vision-based methodology in combination with vehicle-specific power-based energy and emission models is presented. The CalSentry system provides not only typical traffic measures but also gives individual vehicle trajectories (instantaneous dynamics) and recognizes vehicle categories, which are used in the emission models to predict environmental parameters. This estimation process provides far more dynamic and accurate environmental information compared with static emission inventory estimation models.
---
paper_title: Learning, Modeling, and Classification of Vehicle Track Patterns from Live Video

paper_content:

This paper presents two different types of visual activity analysis modules based on vehicle tracking. The highway monitoring module accurately classifies vehicles into eight different types and collects traffic flow statistics by leveraging tracking information. These statistics are continuously accumulated to maintain daily highway models that are used to categorize traffic flow in real time. The path modeling block is a more general analysis tool that learns the normal motions encountered in a scene in an unsupervised fashion. The spatiotemporal motion characteristics of these motion paths are encoded by a hidden Markov model. With the path definitions, abnormal trajectories are detected and future intent is predicted. These modules add realtime situational awareness to highway monitoring for high-level activity and behavior analysis.
---
paper_title: Large Scale Automated Analysis of Vehicle Interactions and Collisions

paper_content:

Road collisions are a worldwide pandemic that can be addressed through the improvement of existing tools for safety analysis. A refined probabilistic framework is presented for the analysis of road-user interactions. In particular, the identification of potential collision points is used to estimate collision probabilities, and their spatial distribution can be visualized. A probabilistic time to collision is introduced, and interactions are grouped into four categories: head-on, rear-end, side, and parallel. The framework is applied to a large data set of video recordings collected in Kentucky that contains more than 300 severe interactions and collisions. The results demonstrate the usefulness of the approach for studying road-user behavior and mechanisms that may lead to collisions.
---
paper_title: Modelling Dynamic Scenes at Unsignalised Road Intersections

paper_content:

Understanding dynamic scenes at road intersections is both crucial and challenging for intelligent vehicles. In order to detect potentially dangerous situations, algorithms are needed that can interpret the behaviour of the actors in the scene and predict its likely evolution. The difficulty of this task arises from the large number of possible scenarios. The conventional answer to this issue is to discard vehicle interactions in the manoeuvre prediction process, i.e. to infer the manoeuvre performed by each vehicle from its past and current behaviour, independently from the other vehicles in the scene. In this paper we show how this affects collision risk estimation in very common scenarios, making it unusable in practice for Advanced Driver Assistance Systems (ADAS) applications. As an alternative we propose a probabilistic model for vehicles traversing unsignalised intersections that accounts for the mutual influence between vehicle manoeuvres. The focus is on the utilisation of contextual information (i.e. layout of the intersection, presence of other vehicles and traffic rules) to interpret a vehicle's behaviour. We show how the model can be used for accurate situation and risk assessment.
---
paper_title: Pedestrian and Bicyclist Intersection Safety Indices: Final Report

paper_content:

The primary objective of this study was to develop safety indices to allow engineers, planners, and other practitioners to proactively prioritize intersection crosswalks and intersection approaches with respect to pedestrian and bicycle safety. The study involved collecting data on pedestrian and bicycle crashes, conflicts, avoidance maneuvers, and subjective ratings of intersection video clips by pedestrian and bicycle experts. There were a total of 68 intersection crosswalks selected for the pedestrian analysis from the cities of Philadelphia, PA; San Jose, CA; Miami-Dade County, FL. The bicycle analysis included 67 intersection approaches from Gainesville, FL; Philadelphia, PA; and Portland and Eugene, OR. Prioritization models were developed based on expert safety ratings and behavioral data. Indicative variables included in the pedestrian safety index model included type or intersection control (signal or stop sign), number of through lanes, 85th percentile vehicle speed, main street traffic volume, and area type. Indicative variables in the bicycle safety models (for through, right-turn, and left-turn bike movements) included various combinations of : presence of bicycle lane, main and cross street traffic volumes, number of through lanes, presence of on-street parking, main street speed limit, presence of traffic signal, number of turn lanes, and others. Through a user-friendly guide, practitioners will be able to use the safety indices to identify which crosswalks and intersection approaches have the highest priority for in-depth pedestrian and bicycle safety evaluations and subsequently use other tools to identify and address potential safety problems.
---
paper_title: Semantic-Based Surveillance Video Retrieval

paper_content:

Visual surveillance produces large amounts of video data. Effective indexing and retrieval from surveillance video databases are very important. Although there are many ways to represent the content of video clips in current video retrieval algorithms, there still exists a semantic gap between users and retrieval systems. Visual surveillance systems supply a platform for investigating semantic-based video retrieval. In this paper, a semantic-based video retrieval framework for visual surveillance is proposed. A cluster-based tracking algorithm is developed to acquire motion trajectories. The trajectories are then clustered hierarchically using the spatial and temporal information, to learn activity models. A hierarchical structure of semantic indexing and retrieval of object activities, where each individual activity automatically inherits all the semantic descriptions of the activity model to which it belongs, is proposed for accessing video clips and individual objects at the semantic level. The proposed retrieval framework supports various queries including queries by keywords, multiple object queries, and queries by sketch. For multiple object queries, succession and simultaneity restrictions, together with depth and breadth first orders, are considered. For sketch-based queries, a method for matching trajectories drawn by users to spatial trajectories is proposed. The effectiveness and efficiency of our framework are tested in a crowded traffic scene
---
paper_title: An Incremental DPMM-Based Method for Trajectory Clustering, Modeling, and Retrieval

paper_content:

Trajectory analysis is the basis for many applications, such as indexing of motion events in videos, activity recognition, and surveillance. In this paper, the Dirichlet process mixture model (DPMM) is applied to trajectory clustering, modeling, and retrieval. We propose an incremental version of a DPMM-based clustering algorithm and apply it to cluster trajectories. An appropriate number of trajectory clusters is determined automatically. When trajectories belonging to new clusters arrive, the new clusters can be identified online and added to the model without any retraining using the previous data. A time-sensitive Dirichlet process mixture model (tDPMM) is applied to each trajectory cluster for learning the trajectory pattern which represents the time-series characteristics of the trajectories in the cluster. Then, a parameterized index is constructed for each cluster. A novel likelihood estimation algorithm for the tDPMM is proposed, and a trajectory-based video retrieval model is developed. The tDPMM-based probabilistic matching method and the DPMM-based model growing method are combined to make the retrieval model scalable and adaptable. Experimental comparisons with state-of-the-art algorithms demonstrate the effectiveness of our algorithm.
---
paper_title: Large Scale Automated Analysis of Vehicle Interactions and Collisions

paper_content:

Road collisions are a worldwide pandemic that can be addressed through the improvement of existing tools for safety analysis. A refined probabilistic framework is presented for the analysis of road-user interactions. In particular, the identification of potential collision points is used to estimate collision probabilities, and their spatial distribution can be visualized. A probabilistic time to collision is introduced, and interactions are grouped into four categories: head-on, rear-end, side, and parallel. The framework is applied to a large data set of video recordings collected in Kentucky that contains more than 300 severe interactions and collisions. The results demonstrate the usefulness of the approach for studying road-user behavior and mechanisms that may lead to collisions.
---
paper_title: Stream-based active unusual event detection

paper_content:

We present a new active learning approach to incorporate human feedback for on-line unusual event detection. In contrast to most existing unsupervised methods that perform passive mining for unusual events, our approach automatically requests supervision for critical points to resolve ambiguities of interest, leading to more robust and accurate detection on subtle unusual events. The active learning strategy is formulated as a stream-based solution, i.e. it makes decision on-the-fly on whether to query for labels. It adaptively combines multiple active learning criteria to achieve (i) quick discovery of unknown event classes and (ii) refinement of classification boundary. Experimental results on busy public space videos show that with minimal human supervision, our approach outperforms existing supervised and unsupervised learning strategies in identifying unusual events. In addition, better performance is achieved by using adaptive multi-criteria approach compared to existing single criterion and multi-criteria active learning strategies.
---
paper_title: Modelling Dynamic Scenes at Unsignalised Road Intersections

paper_content:

Understanding dynamic scenes at road intersections is both crucial and challenging for intelligent vehicles. In order to detect potentially dangerous situations, algorithms are needed that can interpret the behaviour of the actors in the scene and predict its likely evolution. The difficulty of this task arises from the large number of possible scenarios. The conventional answer to this issue is to discard vehicle interactions in the manoeuvre prediction process, i.e. to infer the manoeuvre performed by each vehicle from its past and current behaviour, independently from the other vehicles in the scene. In this paper we show how this affects collision risk estimation in very common scenarios, making it unusable in practice for Advanced Driver Assistance Systems (ADAS) applications. As an alternative we propose a probabilistic model for vehicles traversing unsignalised intersections that accounts for the mutual influence between vehicle manoeuvres. The focus is on the utilisation of contextual information (i.e. layout of the intersection, presence of other vehicles and traffic rules) to interpret a vehicle's behaviour. We show how the model can be used for accurate situation and risk assessment.
---
paper_title: Stream-based joint exploration-exploitation active learning

paper_content:

Learning from streams of evolving and unbounded data is an important problem, for example in visual surveillance or internet scale data. For such large and evolving real-world data, exhaustive supervision is impractical, particularly so when the full space of classes is not known in advance therefore joint class discovery (exploration) and boundary learning (exploitation) becomes critical. Active learning has shown promise in jointly optimising exploration-exploitation with minimal human supervision. However, existing active learning methods either rely on heuristic multi-criteria weighting or are limited to batch processing. In this paper, we present a new unified framework for joint exploration-exploitation active learning in streams without any heuristic weighting. Extensive evaluation on classification of various image and surveillance video datasets demonstrates the superiority of our framework over existing methods.
---
paper_title: Pedestrian and Bicyclist Intersection Safety Indices: Final Report

paper_content:

The primary objective of this study was to develop safety indices to allow engineers, planners, and other practitioners to proactively prioritize intersection crosswalks and intersection approaches with respect to pedestrian and bicycle safety. The study involved collecting data on pedestrian and bicycle crashes, conflicts, avoidance maneuvers, and subjective ratings of intersection video clips by pedestrian and bicycle experts. There were a total of 68 intersection crosswalks selected for the pedestrian analysis from the cities of Philadelphia, PA; San Jose, CA; Miami-Dade County, FL. The bicycle analysis included 67 intersection approaches from Gainesville, FL; Philadelphia, PA; and Portland and Eugene, OR. Prioritization models were developed based on expert safety ratings and behavioral data. Indicative variables included in the pedestrian safety index model included type or intersection control (signal or stop sign), number of through lanes, 85th percentile vehicle speed, main street traffic volume, and area type. Indicative variables in the bicycle safety models (for through, right-turn, and left-turn bike movements) included various combinations of : presence of bicycle lane, main and cross street traffic volumes, number of through lanes, presence of on-street parking, main street speed limit, presence of traffic signal, number of turn lanes, and others. Through a user-friendly guide, practitioners will be able to use the safety indices to identify which crosswalks and intersection approaches have the highest priority for in-depth pedestrian and bicycle safety evaluations and subsequently use other tools to identify and address potential safety problems.
---
paper_title: E3TP: A Novel Trajectory Prediction Algorithm in Moving Objects Databases

paper_content:

Prediction of uncertain trajectories in moving objects databases has recently become a new paradigm for tracking wireless and mobile devices in an accurate and efficient manner, and is critical in law enforcement applications such as criminal tracking analysis. However, existing approaches for prediction in spatio-temporal databases focus on either mining frequent sequential patterns at a certain geographical position, or constructing kinematical models to approximate real-world routes. The former overlooks the fact that movement patterns of objects are most likely to be local, and constrained in some certain region, while the later fails to take into consideration some important factors, e.g., population distribution, and the structure of traffic networks. To cope with those problems, we propose a general trajectory prediction algorithm called E3TP (an Effective, Efficient, and Easy Trajectory Prediction algorithm), which contains four main phases: (i ) mining "hotspot" regions from moving objects databases; (ii ) discovering frequent sequential routes in hotspot areas; (iii ) computing the speed of a variety of moving objects; and (iv ) predicting the dynamic motion behaviors of objects. Experimental results demonstrate that E3TP is an efficient and effective algorithm for trajectory prediction, and the prediction accuracy is about 30% higher than the naive approach. In addition, it is easy-to-use in real-world scenarios.
---
paper_title: Delta-Dual Hierarchical Dirichlet Processes: A pragmatic abnormal behaviour detector

paper_content:

In the security domain a key problem is identifying rare behaviours of interest. Training examples for these behaviours may or may not exist, and if they do exist there will be few examples, quite probably one. We present a novel weakly supervised algorithm that can detect behaviours that either have never before been seen or for which there are few examples. Global context is modelled, allowing the detection of abnormal behaviours that in isolation appear normal. Pragmatic aspects are considered, such that no parameter tuning is required and real time performance is achieved.
---

Title: Understanding vehicular traffic behavior from video: a survey of unsupervised approaches

Section 1: Introduction
Description 1: This section introduces the significance of monitoring road networks using video cameras, the challenges faced with traditional methods, and the shift toward unsupervised machine learning techniques for traffic behavior understanding.

Section 2: Visual Traffic Behavior
Description 2: This section defines traffic behaviors from the perspectives of transportation engineering and computer vision, detailing the complexity of behavior description and the limitations of current video resolutions.

Section 3: Learning Behavior with Trajectories
Description 3: This section discusses trajectory-based learning as a framework for behavior analysis, examining how trajectories are extracted, clustered, and utilized to predict future states and detect anomalies.

Section 4: Clustering Trajectories
Description 4: This section explores various traditional and custom trajectory clustering algorithms, addressing the challenges of comparing trajectories of different lengths and maintaining semantic meaning.

Section 5: Modeling Trajectory Clusters
Description 5: This section describes different methods of modeling trajectory clusters to capture behavior details efficiently, including end-to-end behavior characterization and subtrajectory methods.

Section 6: Discussion of Trajectory Learning
Description 6: This section deliberates on the strengths and weaknesses of trajectory learning, including its dependence on accurate tracking and application-specific behavior definitions.

Section 7: Learning Behavior with Topic Models
Description 7: This section introduces topic models for behavior analysis, their application in video settings, and their ability to learn activities without explicit object tracking.

Section 8: Basic LDA Formulation
Description 8: This section explains the latent Dirichlet allocation model, detailing the process of learning scene behaviors using visual words and topics from video data.

Section 9: Adaptability and Automatically Generating the Number of Topics
Description 9: This section discusses advanced topic models that extend LDA with hierarchical structures and nonparametric priors to automatically determine the number of topics based on training data.

Section 10: Explicitly Modeling Temporal Variations
Description 10: This section covers temporal topic models that capture the dynamic evolution of behaviors over time, including methods like MCTM and DDP-HMM.

Section 11: Incremental Learning and Update
Description 11: This section focuses on the need and techniques for incremental learning to adapt behavior models over long periods, enhancing their practical applicability.

Section 12: Discussion of Topic Models
Description 12: This section reviews the benefits and limitations of topic models in behavior analysis, including their robustness to occlusion and the challenge of parameter selection.

Section 13: Trajectory-topic modeling
Description 13: This section explores the combination of trajectory learning and topic models to jointly target individual behavior patterns and provide a comprehensive understanding of traffic scenes.

Section 14: Computational efficiency
Description 14: This section emphasizes the importance of developing computationally efficient algorithms to handle large datasets and update models effectively for real-time applications.

Section 15: Topic model alternatives
Description 15: This section presents alternative methods to topic models for decoupling tracking from learning processes using other machine learning frameworks.

Section 16: Evaluation and Benchmarking
Description 16: This section outlines available public datasets for behavior analysis, emphasizing the importance of benchmark data for fair comparisons and improvements in research.

Section 17: Evaluation Criteria
Description 17: This section describes methods used to characterize performance in behavior understanding, such as correct clustering rate, completeness, correctness, and ROC curves.

Section 18: Applications
Description 18: This section highlights practical applications of behavior analysis techniques in transportation, such as traffic scene characterization, conflict analysis, motion database query, and abnormality detection.

Section 19: Discussion and Future Directions
Description 19: This section reviews the current state of behavior analysis research, highlighting challenges and future directions for providing nuanced behavior specification and handling large-scale data.

Section 20: Concluding Remarks
Description 20: This section summarizes the survey's findings and emphasizes the importance of unsupervised techniques for understanding traffic behavior from video.