Human Motion Trajectory Prediction: A Survey
---
paper_title: Learning Collective Crowd Behaviors with Dynamic Pedestrian-Agents

paper_content:

Collective behaviors characterize the intrinsic dynamics of the crowds. Automatically understanding collective crowd behaviors has important applications to video surveillance, traffic management and crowd control, while it is closely related to scientific fields such as statistical physics and biology. In this paper, a new mixture model of dynamic pedestrian-Agents (MDA) is proposed to learn the collective behavior patterns of pedestrians in crowded scenes from video sequences. From agent-based modeling, each pedestrian in the crowd is driven by a dynamic pedestrian-agent, which is a linear dynamic system with initial and termination states reflecting the pedestrian's belief of the starting point and the destination. The whole crowd is then modeled as a mixture of dynamic pedestrian-agents. Once the model parameters are learned from the trajectories extracted from videos, MDA can simulate the crowd behaviors. It can also infer the past behaviors and predict the future behaviors of pedestrians given their partially observed trajectories, and classify them different pedestrian behaviors. The effectiveness of MDA and its applications are demonstrated by qualitative and quantitative experiments on various video surveillance sequences.
---
paper_title: Context-based detection of pedestrian crossing intention for autonomous driving in urban environments

paper_content:

This paper focuses on the detection of pedestrian crossing intention to improve the situation awareness for autonomous driving in urban environments. A new definition of pedestrian crossing intention is discussed, which allows self-driving vehicles to identify pedestrians, whose intended actions are relevant for the own behavior planning, at an early stage. We propose a context-based feature descriptor in combination with a SVM classifier for detecting this. The descriptor captures the movement of a pedestrian relative to the road and the spatial layout of other scene elements in a generic manner. The performance of the feature descriptor is evaluated in relation to various SVM setups. Feasibility of the approach is demonstrated with data captured on-board of a vehicle in real inner-city traffic. The evaluation of the classification results confirms, that context-based data is a promising indicator for pedestrian crossing intention and that the proposed feature descriptor is capable of representing this. It is further shown that a lack of information about the pedestrian's posture and body movement results in a delayed detection of the pedestrians changing their crossing intention when compared to a human observer.
---
paper_title: Stereo-Vision-Based Pedestrian's Intention Detection in a Moving Vehicle

paper_content:

We present a method to detect starting, stopping and bending in intentions of pedestrians from a moving vehicle based on stereo-vision. The method focuses on urban scenarios where these pedestrian movements are common and may result in critical situations. Pedestrian intentions are determined by means of an image-based motion contour histogram of oriented gradient descriptor. It is based on silhouettes gathered from stereo data and does not require any compensation of appearance changes resulting from the ego-motion of a vehicle. Nevertheless, it covers small movements indicating a pedestrian's intention. A linear support vector machine with probabilistic estimates is used for classification. We evaluated our method on the publicly available Daimler Pedestrian Path Prediction Benchmark Dataset containing detections of a stateof-the-art pedestrian detector. We detect a pedestrian's stopping intention from 125 ms to 500 ms before standing still within an accuracy range of 80% to 100%. Bending in is detected from 320 ms to 570 ms after a first visible lateral body movement in the same accuracy range. The intention to cross the road from standing still (starting) is detected 250 ms after the first visible motion and, therefore, within the first step with an accuracy of 100%.
---
paper_title: On-Board Detection of Pedestrian Intentions

paper_content:

Avoiding vehicle-to-pedestrian crashes is a critical requirement for nowadays advanced driver assistant systems (ADAS) and future self-driving vehicles. Accordingly, detecting pedestrians from raw sensor data has a history of more than 15 years of research, with vision playing a central role. During the last years, deep learning has boosted the accuracy of image-based pedestrian detectors. However, detection is just the first step towards answering the core question, namely is the vehicle going to crash with a pedestrian provided preventive actions are not taken? Therefore, knowing as soon as possible if a detected pedestrian has the intention of crossing the road ahead of the vehicle is essential for performing safe and comfortable maneuvers that prevent a crash. However, compared to pedestrian detection, there is relatively little literature on detecting pedestrian intentions. This paper aims to contribute along this line by presenting a new vision-based approach which analyzes the pose of a pedestrian along several frames to determine if he or she is going to enter the road or not. We present experiments showing 750 ms of anticipation for pedestrians crossing the road, which at a typical urban driving speed of 50 km/h can provide 15 additional meters (compared to a pure pedestrian detector) for vehicle automatic reactions or to warn the driver. Moreover, in contrast with state-of-the-art methods, our approach is monocular, neither requiring stereo nor optical flow information.
---
paper_title: Context-Based Path Prediction for Targets with Switching Dynamics

paper_content:

Anticipating future situations from streaming sensor data is a key perception challenge for mobile robotics and automated vehicles. We address the problem of predicting the path of objects with multiple dynamic modes. The dynamics of such targets can be described by a Switching Linear Dynamical System (SLDS). However, predictions from this probabilistic model cannot anticipate when a change in dynamic mode will occur. We propose to extract various types of cues with computer vision to provide context on the target’s behavior, and incorporate these in a Dynamic Bayesian Network (DBN). The DBN extends the SLDS by conditioning the mode transition probabilities on additional context states. We describe efficient online inference in this DBN for probabilistic path prediction, accounting for uncertainty in both measurements and target behavior. Our approach is illustrated on two scenarios in the Intelligent Vehicles domain concerning pedestrians and cyclists, so-called Vulnerable Road Users (VRUs). Here, context cues include the static environment of the VRU, its dynamic environment, and its observed actions. Experiments using stereo vision data from a moving vehicle demonstrate that the proposed approach results in more accurate path prediction than SLDS at the relevant short time horizon (1 s). It slightly outperforms a computationally more demanding state-of-the-art method.
---
paper_title: Pedestrian Path Prediction with Recursive Bayesian Filters: A Comparative Study

paper_content:

In the context of intelligent vehicles, we perform a comparative study on recursive Bayesian filters for pedestrian path prediction at short time horizons (< 2s). We consider Extended Kalman Filters (EKF) based on single dynamical models and Interacting Multiple Models (IMM) combining several such basic models (constant velocity/acceleration/turn). These are applied to four typical pedestrian motion types (crossing, stopping, bending in, starting). Position measurements are provided by an external state-of-the-art stereo vision-based pedestrian detector. We investigate the accuracy of position estimation and path prediction, and the benefit of the IMMs vs. the simpler single dynamical models. Special care is given to the proper sensor modeling and parameter optimization. The dataset and evaluation framework are made public to facilitate benchmarking.
---
paper_title: Using road topology to improve cyclist path prediction

paper_content:

We learn motion models for cyclist path prediction on real-world tracks obtained from a moving vehicle, and propose to exploit the local road topology to obtain better predictive distributions. The tracks are extracted from the Tsinghua-Daimler Cyclist Benchmark for cyclist detection, and corrected for vehicle egomotion. Tracks are then spatially aligned to local curves and crossings in the road. We study a standard approach for path prediction in the literature based on Kalman Filters, as well as a mixture of specialized filters related to specific road orientations at junctions. Our experiments demonstrate an improved prediction accuracy (up to 20% on sharp turns) of mixing specialized motion models for canonical directions, and prior knowledge on the road topology. The new track data complements the existing video, disparity and annotation data of the original benchmark, and will be made publicly available.
---
paper_title: Will the Pedestrian Cross? A Study on Pedestrian Path Prediction

paper_content:

Future vehicle systems for active pedestrian safety will not only require a high recognition performance but also an accurate analysis of the developing traffic situation. In this paper, we present a study on pedestrian path prediction and action classification at short subsecond time intervals. We consider four representative approaches: two novel approaches (based on Gaussian process dynamical models and probabilistic hierarchical trajectory matching) that use augmented features derived from dense optical flow and two approaches as baseline that use positional information only (a Kalman filter and its extension to interacting multiple models). In experiments using stereo vision data obtained from a vehicle, we investigate the accuracy of path prediction and action classification at various time horizons, the effect of various errors (image localization, vehicle egomotion estimation), and the benefit of the proposed approaches. The scenario of interest is that of a crossing pedestrian, who might stop or continue walking at the road curbside. Results indicate similar performance of the four approaches on walking motion, with near-linear dynamics. During stopping, however, the two newly proposed approaches, with nonlinear and/or higher order models and augmented motion features, achieve a more accurate position prediction of 10-50 cm at a time horizon of 0-0.77 s around the stopping event.
---
paper_title: Behavioral Priors for Detection and Tracking of Pedestrians in Video Sequences

paper_content:

In this paper we address the problem of detection and tracking of pedestrians in complex scenarios. The inclusion of prior knowledge is more and more crucial in scene analysis to guarantee flexibility and robustness, necessary to have reliability in complex scenes. We aim to combine image processing methods with behavioral models of pedestrian dynamics, calibrated on real data. We introduce Discrete Choice Models (DCM) for pedestrian behavior and we discuss their integration in a detection and tracking context. The obtained results show how it is possible to combine both methodologies to improve the performances of such systems in complex sequences.
---
paper_title: Incremental learning of human social behaviors with feature-based spatial effects

paper_content:

In the near future, robots will exist in our home, office, hospital, school and all other public places. In other words, robots have to work and share environments with human. However, there exist many invisible social rules or social protocols in human society. For example, people usually stand in a line to access certain services. It seems that there is a social force or a “spatial effect” which forces people to perform the behavior of standing in a line. Unfortunately, these spatial effects are usually invisible and immeasurable. It makes social behavior understanding to be a difficult problem. The robot that does not comprehend these spatial effects might harm people or itself. Previous methods of behavior understanding mainly only targeted certain scenarios or environments. This paper follows the framework of SBCM (Spatial Behavior Cognition Model)[3] and extends to feature-based spatial effects. By utilizing feature to represent the spatial effect, it is easier to apply known spatial effects to different environments. This paper also demonstrates that the robot is able to learn and execute some common social behaviors under the proposed method. Several simulations and experiments are conducted to verify the proposed idea in this paper.
---
paper_title: Visual Path Prediction in Complex Scenes with Crowded Moving Objects

paper_content:

This paper proposes a novel path prediction algorithm for progressing one step further than the existing works focusing on single target path prediction. In this paper, we consider moving dynamics of co-occurring objects for path prediction in a scene that includes crowded moving objects. To solve this problem, we first suggest a two-layered probabilistic model to find major movement patterns and their cooccurrence tendency. By utilizing the unsupervised learning results from the model, we present an algorithm to find the future location of any target object. Through extensive qualitative/quantitative experiments, we show that our algorithm can find a plausible future path in complex scenes with a large number of moving objects.
---
paper_title: Bi-Prediction: Pedestrian Trajectory Prediction Based on Bidirectional LSTM Classification

paper_content:

Pedestrian trajectory prediction is important in various applications such as driverless vehicles, social robots, intelligent tracking systems and space planning. Existing methods focus on analysing the influence of neighbours but ignore the effect of the intended destinations of pedestrians which also plays a key role in route planning. In this paper, we propose a novel two- stage trajectory prediction method to yield multiple prediction trajectories with different probabilities towards different destination regions in the scene. Our method, which we refer to as Bi-Prediction, uses a bidirectional LSTM architecture to automatically classify trajectories into a small number of route classes before trajectory prediction. We have evaluated our method against two baseline methods and three state-of-art methods on two benchmark datasets. Our experimental results show that the extra classification stage improves the accuracy of the predicted trajectories.
---
paper_title: Modelling pedestrian trajectory patterns with Gaussian processes

paper_content:

We propose a non-parametric model for pedestrian motion based on Gaussian Process regression, in which trajectory data are modelled by regressing relative motion against current position. We show how the underlying model can be learned in an unsupervised fashion, demonstrating this on two databases collected from static surveillance cameras. We furthermore exemplify the use of model for prediction, comparing the recently proposed GP-Bayesfilters with a Monte Carlo method. We illustrate the benefit of this approach for long term motion prediction where parametric models such as Kalman Filters would perform poorly.
---
paper_title: Forecasting Interactive Dynamics of Pedestrians with Fictitious Play

paper_content:

We develop predictive models of pedestrian dynamics by encoding the coupled nature of multi-pedestrian interaction using game theory, and deep learning-based visual analysis to estimate person-specific behavior parameters. Building predictive models for multi-pedestrian interactions however, is very challenging due to two reasons: (1) the dynamics of interaction are complex interdependent processes, where the predicted behavior of one pedestrian can affect the actions taken by others and (2) dynamics are variable depending on an individuals physical characteristics (e.g., an older person may walk slowly while the younger person may walk faster). To address these challenges, we (1) utilize concepts from game theory to model the interdependent decision making process of multiple pedestrians and (2) use visual classifiers to learn a mapping from pedestrian appearance to behavior parameters. We evaluate our proposed model on several public multiple pedestrian interaction video datasets. Results show that our strategic planning model explains human interactions 25% better when compared to state-of-the-art methods.
---
paper_title: Modelling Smooth Paths Using Gaussian Processes

paper_content:

A generative model based on the gaussian mixture model and gaussian processes is presented in this paper. Typical motion paths are learnt and then used for motion prediction using this model. The principal novel aspect of this approach is the modelling of paths using gaussian processes. It allows the representation of smooth trajectories and avoids discretization problems found in most existing methods. Gaussian processes not only provides a comprehensive and formal theoretical framework to work with, it also lends itself naturally to path clustering using gaussian mixture models. Learning is performed using expectation maximization where the E-Step uses variational methods to maximize its lower bound before optimization over parameters are performed in the M-Step.
---
paper_title: Pedestrian Trajectory Prediction in Extremely Crowded Scenarios

paper_content:

Pedestrian trajectory prediction under crowded circumstances is a challenging problem owing to human interaction and the complexity of the trajectory pattern. Various methods have been proposed for solving this problem, ranging from traditional Bayesian analysis to Social Force model and deep learning methods. However, most existing models heavily depend on specific scenarios because the trajectory model is constructed in absolute coordinates even though the motion trajectory as well as human interaction are in relative motion. In this study, a novel trajectory prediction model is proposed to capture the relative motion of pedestrians in extremely crowded scenarios. Trajectory sequences and human interaction are first represented with relative motion and then integrated to our model to predict pedestrians’ trajectories. The proposed model is based on Long Short Term Memory (LSTM) structure and consists of an encoder and a decoder which are trained by truncated back propagation. In addition, an anisotropic neighborhood setting is proposed instead of traditional neighborhood analysis. The proposed approach is validated using trajectory data acquired at an extremely crowded train station in Tokyo, Japan. The trajectory prediction experiments demonstrated that the proposed method outperforms existing methods and is stable for predictions of varying length even when the model is trained with a controlled short trajectory sequence.
---
paper_title: GLMP- realtime pedestrian path prediction using global and local movement patterns

paper_content:

We present a novel real-time algorithm to predict the path of pedestrians in cluttered environments. Our approach makes no assumption about pedestrian motion or crowd density, and is useful for short-term as well as long-term prediction. We interactively learn the characteristics of pedestrian motion and movement patterns from 2D trajectories using Bayesian inference. These include local movement patterns corresponding to the current and preferred velocities and global characteristics such as entry points and movement features. Our approach involves no precomputation and we demonstrate the real-time performance of our prediction algorithm on sparse and noisy trajectory data extracted from dense indoor and outdoor crowd videos. The combination of local and global movement patterns can improve the accuracy of long-term prediction by 12–18% over prior methods in high-density videos.
---
paper_title: Goal-directed pedestrian model for long-term motion prediction with application to robot motion planning

paper_content:

A probabilistic goal-directed model is proposed for pedestrian motion using navigation function and statistics of human motion gathered in the environment. In comparison with existing models, this model is both computationally inexpensive and does not fail when the optimal direction of motion in terms of this model is non-unique. We further introduce a Rapidly-Exploring Random Tree (RRT)-based path planner developed for planning in state-time space. With the help of an improved distance metric, the planner is much faster than RRT-Blossom [11] in complex maps. In an environment with 10 pedestrians, the planner and motion prediction combined can perform in near-real time.
---
paper_title: SS-LSTM: A Hierarchical LSTM Model for Pedestrian Trajectory Prediction

paper_content:

Pedestrian trajectory prediction is an extremely challenging problem because of the crowdedness and clutter of the scenes. Previous deep learning LSTM-based approaches focus on the neighbourhood influence of pedestrians but ignore the scene layouts in pedestrian trajectory prediction. In this paper, a novel hierarchical LSTM-based network is proposed to consider both the influence of social neighbourhood and scene layouts. Our SS-LSTM, which stands for Social-Scene-LSTM, uses three different LSTMs to capture person, social and scene scale information. We also use a circular shape neighbourhood setting instead of the traditional rectangular shape neighbourhood in the social scale. We evaluate our proposed method against two baseline methods and a state-of-art technique on three public datasets. The results show that our method outperforms other methods and that using circular shape neighbourhood improves the prediction accuracy.
---
paper_title: Learning Collective Crowd Behaviors with Dynamic Pedestrian-Agents

paper_content:

Collective behaviors characterize the intrinsic dynamics of the crowds. Automatically understanding collective crowd behaviors has important applications to video surveillance, traffic management and crowd control, while it is closely related to scientific fields such as statistical physics and biology. In this paper, a new mixture model of dynamic pedestrian-Agents (MDA) is proposed to learn the collective behavior patterns of pedestrians in crowded scenes from video sequences. From agent-based modeling, each pedestrian in the crowd is driven by a dynamic pedestrian-agent, which is a linear dynamic system with initial and termination states reflecting the pedestrian's belief of the starting point and the destination. The whole crowd is then modeled as a mixture of dynamic pedestrian-agents. Once the model parameters are learned from the trajectories extracted from videos, MDA can simulate the crowd behaviors. It can also infer the past behaviors and predict the future behaviors of pedestrians given their partially observed trajectories, and classify them different pedestrian behaviors. The effectiveness of MDA and its applications are demonstrated by qualitative and quantitative experiments on various video surveillance sequences.
---
paper_title: Gaussian process regression flow for analysis of motion trajectories

paper_content:

Recognition of motions and activities of objects in videos requires effective representations for analysis and matching of motion trajectories. In this paper, we introduce a new representation specifically aimed at matching motion trajectories. We model a trajectory as a continuous dense flow field from a sparse set of vector sequences using Gaussian Process Regression. Furthermore, we introduce a random sampling strategy for learning stable classes of motions from limited data. Our representation allows for incrementally predicting possible paths and detecting anomalous events from online trajectories. This representation also supports matching of complex motions with acceleration changes and pauses or stops within a trajectory. We use the proposed approach for classifying and predicting motion trajectories in traffic monitoring domains and test on several data sets. We show that our approach works well on various types of complete and incomplete trajectories from a variety of video data sets with different frame rates.
---
paper_title: You'll never walk alone: Modeling social behavior for multi-target tracking

paper_content:

Object tracking typically relies on a dynamic model to predict the object's location from its past trajectory. In crowded scenarios a strong dynamic model is particularly important, because more accurate predictions allow for smaller search regions, which greatly simplifies data association. Traditional dynamic models predict the location for each target solely based on its own history, without taking into account the remaining scene objects. Collisions are resolved only when they happen. Such an approach ignores important aspects of human behavior: people are driven by their future destination, take into account their environment, anticipate collisions, and adjust their trajectories at an early stage in order to avoid them. In this work, we introduce a model of dynamic social behavior, inspired by models developed for crowd simulation. The model is trained with videos recorded from birds-eye view at busy locations, and applied as a motion model for multi-people tracking from a vehicle-mounted camera. Experiments on real sequences show that accounting for social interactions and scene knowledge improves tracking performance, especially during occlusions.
---
paper_title: A survey on motion prediction and risk assessment for intelligent vehicles

paper_content:

With the objective to improve road safety, the automotive industry is moving toward more “intelligent” vehicles. One of the major challenges is to detect dangerous situations and react accordingly in order to avoid or mitigate accidents. This requires predicting the likely evolution of the current traffic situation, and assessing how dangerous that future situation might be. This paper is a survey of existing methods for motion prediction and risk assessment for intelligent vehicles. The proposed classification is based on the semantics used to define motion and risk. We point out the tradeoff between model completeness and real-time constraints, and the fact that the choice of a risk assessment method is influenced by the selected motion model.
---
paper_title: Survey on Vision-based Path Prediction

paper_content:

Path prediction is a fundamental task for estimating how pedestrians or vehicles are going to move in a scene. Because path prediction as a task of computer vision uses video as input, various information used for prediction, such as the environment surrounding the target and the internal state of the target, need to be estimated from the video in addition to predicting paths. Many prediction approaches that include understanding the environment and the internal state have been proposed. In this survey, we systematically summarize methods of path prediction that take video as input and extract features from the video. Moreover, we introduce datasets used to evaluate path prediction methods quantitatively.
---
paper_title: Comparison and evaluation of pedestrian motion models for vehicle safety systems

paper_content:

In this paper the influence of different information on the performance of pedestrian motion models is evaluated. For this, besides of dynamic and physiological constraints, additional information as head orientation, typical pedestrian motion behavior and environment informations are used. A vehicle sensor system, consisting of a 12Mpx camera and Lidar provides the pedestrian and environment information in the considered test cases. Based on this, four pedestrian models, each based on different information, are used to predict the future position of the pedestrian. The predicted area is presented by a probability grid. The results show the performance of the models for evaluation parameters as prediction error, standard deviation and collision probability. Earlier works show, that the models have different performance depending on the prediction time. So a new introduced system fuses the probability map of each model to achieve a continuous prediction performance.
---
paper_title: Predicting and recognizing human interactions in public spaces

paper_content:

We present an extensive survey of methods for recognizing human interactions and propose a method for predicting rendezvous areas in observable and unobservable regions using sparse motion information. Rendezvous areas indicate where people are likely to interact with each other or with static objects (e.g., a door, an information desk or a meeting point). The proposed method infers the direction of movement by calculating prediction lines from displacement vectors and temporally accumulates intersecting locations generated by prediction lines. The intersections are then used as candidate rendezvous areas and modeled as spatial probability density functions using Gaussian Mixture Models. We validate the proposed method to predict dynamic and static rendezvous areas on real-world datasets and compare it with related approaches.
---
paper_title: Human-aware robot navigation: A survey

paper_content:

Navigation is a basic skill for autonomous robots. In the last years human-robot interaction has become an important research field that spans all of the robot capabilities including perception, reasoning, learning, manipulation and navigation. For navigation, the presence of humans requires novel approaches that take into account the constraints of human comfort as well as social rules. Besides these constraints, putting robots among humans opens new interaction possibilities for robots, also for navigation tasks, such as robot guides. This paper provides a survey of existing approaches to human-aware navigation and offers a general classification scheme for the presented methods.
---
paper_title: A Review of Social-Aware Navigation Frameworks for Service Robot in Dynamic Human Environments

paper_content:

The emergence of service robot into human daily life in the past years has opened up various challenges including human-robot interaction, joint-goal achievement and machine learning. Social-aware navigation also gains vast research attention in enhancing the social capabilities of service robots. Human motions are stochastic and social conventions are very complex. Sophisticated approaches are needed for a robot to abide to these social rules and perform obstacle avoidance. To maintain the level of social comfort and achieve a given task, the robot navigation is now no longer a search for a shortest collision-free path, but a multi-objective problem that requires a unified social-aware navigation framework. A careful selection of navigation components including global planner, local planner, the prediction model and a suitable robot platform is also required to offer an effective navigation amidst the dynamic human environment. Hence, this review paper aims to offer insights for service robot implementation by highlighting four varieties of navigation frameworks, various navigation components and different robot platforms.
---
paper_title: A Literature Review on the Prediction of Pedestrian Behavior in Urban Scenarios

paper_content:

The ability to anticipate pedestrian actions on streets is a safety issue for intelligent cars and has increasingly drawn the attention of the automotive industry. Estimating when pedestrians will cross streets has proved a challenging task, since they can move in many different directions, suddenly change motion, be occluded by a variety of obstacles and distracted while talking to other pedestrians or typing on a mobile phone. Moreover, their decisions can also be affected by several factors. This paper explores the ways pedestrians' intention estimation has been studied, evaluated, and evolved. It provides a literature review on pedestrian behavior prediction, addresses available solutions, state-of-the-art developments, and hurdles to be overcome towards reaching a solution that is closer to the human ability to predict and interpret such scenarios. Although many studies can precisely estimate pedestrians' positioning one second before they cross a street, most of them cannot precisely predict when they will stop at a curb.
---
paper_title: How good is my prediction? Finding a similarity measure for trajectory prediction evaluation

paper_content:

The reliable prediction of traffic participants' trajectories is an important challenge for automated driving. Prediction methods that try to deal with this challenge need similarity measures for trajectories in order to evaluate the quality of their prediction. Currently there exists no commonly accepted similarity measure suitable for this task. In this paper we review common trajectory similarity measures and analyze them with regard to prediction evaluation. Further we introduce a new approach for synthesizing a hybrid measure that combines a set of similarity measures and provide a heuristic to determine the parameters for this approach.
---
paper_title: Prediction of moving objects in dynamic environments using Kalman filters

paper_content:

In this work, we describe a framework for predicting future positions and orientation of moving obstacles in a time-varying environment using Kalman filtering techniques. No constraint are placed on the obstacles motion. The proposed algorithm can be used in a variety of applications, one of which is robot motion planning in time varying environments. The advantage of using this model compared to reported ones in the literature is it's ability to start the prediction process from the first time step without the need to wait for few time steps before starting the prediction process. Early experimental results are very encouraging.
---
paper_title: A probabilistic model for estimating driver behaviors and vehicle trajectories in traffic environments

paper_content:

This paper presents a filter that is able to simultaneously estimate the behaviors of traffic participants and anticipate their future trajectories. This is achieved by recognizing the type of situation derived from the local situational context, which subsumes all information relevant for the drivers decision making. By explicitly taking into account the interactions between vehicles, it achieves a comprehensive situational understanding, inevitable for autonomous vehicles and driver assistance systems. This provides the necessary information for safe behavior decision making or motion planning. The filter is modeled as a Dynamic Bayesian Network. The factored state space, modeling the causal dependencies, allows to describe the models in a compact fashion and reduces the computational complexity of the inference process. The filter is evaluated in the context of a highway scenario, showing a good performance even with very noisy measurements. The presented framework is intended to be used in traffic environments but can be easily transferred to other robotic domains.
---
paper_title: Mobile Agent Trajectory Prediction using Bayesian Nonparametric Reachability Trees

paper_content:

This paper presents an efficient trajectory prediction algorithm that has been developed to improve the performance of future collision avoidance and detection systems. The main idea is to embed the inferred intention information of surrounding agents into their estimated reachability sets to obtain a probabilistic description of their future paths. More specifically, the proposed approach combines the recently developed RRT-Reach algorithm and mixtures of Gaussian Processes. RRT-Reach was introduced by the authors as an extension of the closed-loop rapidly-exploring random tree (CL-RRT) algorithm to compute reachable sets of moving objects in real-time. A mixture of Gaussian processes (GP) is a flexible nonparametric Bayesian model used to represent a distribution over trajectories and have been previously demonstrated by the authors in a UAV interception and tracking of ground vehicles planning scheme. The mixture is trained using typical maneuvers learned from statistical data, and RRT-Reach utilizes samples from the GP to grow probabilistically weighted feasible paths of the surrounding vehicles. The resulting approach, denoted as RR-GP, has RRTReach’s benefits of computing trajectories that are dynamically feasible by construction, therefore efficiently approximating the reachability set of surrounding vehicles following typical patterns. RRT-GP also features the GP mixture’s benefits of providing a probabilistic weighting on the feasible trajectories produced by RRTReach, allowing our system to systematically weight trajectories by their likelihood. A demonstrative example on a car-like vehicle illustrates the advantages of the RR-GP approach by comparing it to two other GP-based algorithms.
---
paper_title: Trajectory prediction of cyclists using a physical model and an artificial neural network

paper_content:

This article presents two methods for predicting the trajectories of cyclists at an intersection and compares them to a Kalman Filter (KF) approach. The first method uses a physical model of cyclists to predict their future position. The second method is based on a polynomial least-squares approximation in combination with a multilayer perceptron artificial neural network and is able to predict the future position of cyclists independent of their motion type such as “Starting”, “Stopping”, “Waiting” or “Passing”. To evaluate the performance of the methods, 566 tracks (394 for training, 172 for testing) of uninstructed cyclists were recorded at a public intersection using a wide angle stereo camera system and laser scanners. Using the tracks as input data, the future trajectory was predicted for a time horizon of 2.5 s. For starting motions the prediction using the physical model leads to 27% more accurate positions than the KF approach for a forecast horizon of 2.5 s. The neural network shows a 34% more accurate result for starting and stopping motions and a similar result for waiting and passing motions.
---
paper_title: Reachability analysis of nonlinear systems with uncertain parameters using conservative linearization

paper_content:

Given an initial set of a nonlinear system with uncertain parameters and inputs, the set of states that can possibly be reached is computed. The approach is based on local linearizations of the nonlinear system, while linearization errors are considered by Lagrange remainders. These errors are added as uncertain inputs, such that the reachable set of the locally linearized system encloses the one of the original system. The linearization error is controlled by splitting of reachable sets. Reachable sets are represented by zonotopes, allowing an efficient computation in relatively high-dimensional space.
---
paper_title: People tracking with human motion predictions from social forces

paper_content:

For many tasks in populated environments, robots need to keep track of current and future motion states of people. Most approaches to people tracking make weak assumptions on human motion such as constant velocity or acceleration. But even over a short period, human behavior is more complex and influenced by factors such as the intended goal, other people, objects in the environment, and social rules. This motivates the use of more sophisticated motion models for people tracking especially since humans frequently undergo lengthy occlusion events. In this paper, we consider computational models developed in the cognitive and social science communities that describe individual and collective pedestrian dynamics for tasks such as crowd behavior analysis. In particular, we integrate a model based on a social force concept into a multi-hypothesis target tracker. We show how the refined motion predictions translate into more informed probability distributions over hypotheses and finally into a more robust tracking behavior and better occlusion handling. In experiments in indoor and outdoor environments with data from a laser range finder, the social force model leads to more accurate tracking with up to two times fewer data association errors.
---
paper_title: Unfreezing the robot: Navigation in dense, interacting crowds

paper_content:

In this paper, we study the safe navigation of a mobile robot through crowds of dynamic agents with uncertain trajectories. Existing algorithms suffer from the “freezing robot” problem: once the environment surpasses a certain level of complexity, the planner decides that all forward paths are unsafe, and the robot freezes in place (or performs unnecessary maneuvers) to avoid collisions. Since a feasible path typically exists, this behavior is suboptimal. Existing approaches have focused on reducing the predictive uncertainty for individual agents by employing more informed models or heuristically limiting the predictive covariance to prevent this overcautious behavior. In this work, we demonstrate that both the individual prediction and the predictive uncertainty have little to do with the frozen robot problem. Our key insight is that dynamic agents solve the frozen robot problem by engaging in “joint collision avoidance”: They cooperatively make room to create feasible trajectories. We develop IGP, a nonparametric statistical model based on dependent output Gaussian processes that can estimate crowd interaction from data. Our model naturally captures the non-Markov nature of agent trajectories, as well as their goal-driven navigation. We then show how planning in this model can be efficiently implemented using particle based inference. Lastly, we evaluate our model on a dataset of pedestrians entering and leaving a building, first comparing the model with actual pedestrians, and find that the algorithm either outperforms human pedestrians or performs very similarly to the pedestrians. We also present an experiment where a covariance reduction method results in highly overcautious behavior, while our model performs desirably.
---
paper_title: Incremental learning of human social behaviors with feature-based spatial effects

paper_content:

In the near future, robots will exist in our home, office, hospital, school and all other public places. In other words, robots have to work and share environments with human. However, there exist many invisible social rules or social protocols in human society. For example, people usually stand in a line to access certain services. It seems that there is a social force or a “spatial effect” which forces people to perform the behavior of standing in a line. Unfortunately, these spatial effects are usually invisible and immeasurable. It makes social behavior understanding to be a difficult problem. The robot that does not comprehend these spatial effects might harm people or itself. Previous methods of behavior understanding mainly only targeted certain scenarios or environments. This paper follows the framework of SBCM (Spatial Behavior Cognition Model)[3] and extends to feature-based spatial effects. By utilizing feature to represent the spatial effect, it is easier to apply known spatial effects to different environments. This paper also demonstrates that the robot is able to learn and execute some common social behaviors under the proposed method. Several simulations and experiments are conducted to verify the proposed idea in this paper.
---
paper_title: Context-Based Path Prediction for Targets with Switching Dynamics

paper_content:

Anticipating future situations from streaming sensor data is a key perception challenge for mobile robotics and automated vehicles. We address the problem of predicting the path of objects with multiple dynamic modes. The dynamics of such targets can be described by a Switching Linear Dynamical System (SLDS). However, predictions from this probabilistic model cannot anticipate when a change in dynamic mode will occur. We propose to extract various types of cues with computer vision to provide context on the target’s behavior, and incorporate these in a Dynamic Bayesian Network (DBN). The DBN extends the SLDS by conditioning the mode transition probabilities on additional context states. We describe efficient online inference in this DBN for probabilistic path prediction, accounting for uncertainty in both measurements and target behavior. Our approach is illustrated on two scenarios in the Intelligent Vehicles domain concerning pedestrians and cyclists, so-called Vulnerable Road Users (VRUs). Here, context cues include the static environment of the VRU, its dynamic environment, and its observed actions. Experiments using stereo vision data from a moving vehicle demonstrate that the proposed approach results in more accurate path prediction than SLDS at the relevant short time horizon (1 s). It slightly outperforms a computationally more demanding state-of-the-art method.
---
paper_title: Voronoi tracking: location estimation using sparse and noisy sensor data

paper_content:

Tracking the activity of people in indoor environments has gained considerable attention in the robotics community over the last years. Most of the existing approaches are based on sensors, which allow to accurately determining the locations of people but do not provide means to distinguish between different persons. In this paper we propose a novel approach to tracking moving objects and their identity using noisy, sparse information collected by id-sensors such as infrared and ultrasound badge systems. The key idea of our approach is to use particle filters to estimate the locations of people on the Voronoi graph of the environment. By restricting particles to a graph, we make use of the inherent structure of indoor environments. The approach has two key advantages. First, it is by far more efficient and robust than unconstrained particle filters. Second, the Voronoi graph provides a natural discretization of human motion, which allows us to apply unsupervised learning techniques to derive typical motion patterns of the people in the environment. Experiments using a robot to collect ground-truth data indicate the superior performance of Voronoi tracking. Furthermore, we demonstrate that EM-based learning of behavior patterns increases the tracking performance and provides valuable information for high-level behavior recognition.
---
paper_title: Planning-based prediction for pedestrians

paper_content:

We present a novel approach for determining robot movements that efficiently accomplish the robot's tasks while not hindering the movements of people within the environment. Our approach models the goal-directed trajectories of pedestrians using maximum entropy inverse optimal control. The advantage of this modeling approach is the generality of its learned cost function to changes in the environment and to entirely different environments. We employ the predictions of this model of pedestrian trajectories in a novel incremental planner and quantitatively show the improvement in hindrance-sensitive robot trajectory planning provided by our approach.
---
paper_title: Enabling Flow Awareness for Mobile Robots in Partially Observable Environments

paper_content:

Understanding the environment is a key requirement for any autonomous robot operation. There is extensive research on mapping geometric structure and perceiving objects. However, the environment is ...
---
paper_title: Transferable Pedestrian Motion Prediction Models at Intersections

paper_content:

One desirable capability of autonomous cars is to accurately predict the pedestrian motion near intersections for safe and efficient trajectory planning. We are interested in developing transfer learning algorithms that can be trained on the pedestrian trajectories collected at one intersection and yet still provide accurate predictions of the trajectories at another, previously unseen intersection. We first discussed the feature selection for transferable pedestrian motion models in general. Following this discussion, we developed one transferable pedestrian motion prediction algorithm based on Inverse Reinforcement Learning (IRL) that infers pedestrian intentions and predicts future trajectories based on observed trajectory. We evaluated our algorithm on a dataset collected at two intersections, trained at one intersection and tested at the other intersection. We used the accuracy of augmented semi-nonnegative sparse coding (ASNSC), trained and tested at the same intersection as a baseline. The result shows that the proposed algorithm improves the baseline accuracy by 40% in the non-transfer task, and 16% in the transfer task.
---
paper_title: Using road topology to improve cyclist path prediction

paper_content:

We learn motion models for cyclist path prediction on real-world tracks obtained from a moving vehicle, and propose to exploit the local road topology to obtain better predictive distributions. The tracks are extracted from the Tsinghua-Daimler Cyclist Benchmark for cyclist detection, and corrected for vehicle egomotion. Tracks are then spatially aligned to local curves and crossings in the road. We study a standard approach for path prediction in the literature based on Kalman Filters, as well as a mixture of specialized filters related to specific road orientations at junctions. Our experiments demonstrate an improved prediction accuracy (up to 20% on sharp turns) of mixing specialized motion models for canonical directions, and prior knowledge on the road topology. The new track data complements the existing video, disparity and annotation data of the original benchmark, and will be made publicly available.
---
paper_title: Unsupervised robot learning to predict person motion

paper_content:

Socially interacting robots will need to understand the intentions and recognize the behaviors of people they come in contact with. In this paper we look at how a robot can learn to recognize and predict people's intended path based on its own observations of people over time. Our approach uses people tracking on the robot from either RGBD cameras or LIDAR. The tracks are separated into homogeneous motion classes using a pre-trained SVM. Then the individual classes are clustered and prototypes are extracted from each cluster. These are then used to predict a person's future motion based on matching to a partial prototype and using the rest of the prototype as the predicted motion. Results from experiments in a kitchen environment in our lab demonstrate the capabilities of the proposed method.
---
paper_title: Intent-aware long-term prediction of pedestrian motion

paper_content:

We present a method to predict long-term motion of pedestrians, modeling their behavior as jump-Markov processes with their goal a hidden variable. Assuming approximately rational behavior, and incorporating environmental constraints and biases, including time-varying ones imposed by traffic lights, we model intent as a policy in a Markov decision process framework. We infer pedestrian state using a Rao-Blackwellized filter, and intent by planning according to a stochastic policy, reflecting individual preferences in aiming at the same goal.
---
paper_title: Who are you with and where are you going?

paper_content:

We propose an agent-based behavioral model of pedestrians to improve tracking performance in realistic scenarios. In this model, we view pedestrians as decision-making agents who consider a plethora of personal, social, and environmental factors to decide where to go next. We formulate prediction of pedestrian behavior as an energy minimization on this model. Two of our main contributions are simple, yet effective estimates of pedestrian destination and social relationships (groups). Our final contribution is to incorporate these hidden properties into an energy formulation that results in accurate behavioral prediction. We evaluate both our estimates of destination and grouping, as well as our accuracy at prediction and tracking against state of the art behavioral model and show improvements, especially in the challenging observational situation of infrequent appearance observations–something that might occur in thousands of webcams available on the Internet.
---
paper_title: Long-term Path Prediction in Urban Scenarios using Circular Distributions

paper_content:

Abstract Human ability to foresee the near future plays a key role in everyone's life to prevent potentially dangerous situations. To be able to make predictions is crucial when people have to interact with the surrounding environment. Modeling such capability can lead to the design of automated warning systems and provide moving robots with an intelligent way of interaction with changing situation. In this work we focus on a typical urban human-scene where we aim at predicting an agent's behavior using a stochastic model. In this approach, we fuse the various factors that would contribute to a human motion in different contexts. Our method uses previously observed trajectories to build point-wise circular distributions that after combination, provide a statistical smooth prediction towards the most likely areas. More specifically, a ray-launching procedure, based on a semantic segmentation, gives a coarse scene representation for collision avoidance; a nearly-constant velocity dynamic model smooths the acceleration progression and knowledge of the agent's destination may further steer the path prediction. Experimental results in structured scenes, validate the effectiveness of the method in predicting paths in comparison to actual trajectories.
---
paper_title: Pedestrian's Trajectory Forecast in Public Traffic with Artificial Neural Networks

paper_content:

This paper focuses on forecasting of pedestrian's short-time trajectories up to 2.5 s for traffic safety applications. We present a self-learning approach based on artificial neural network movement models and compare it to traditional constant velocity Kalman Filter prediction and extrapolation of polynomials fitted using a least-squares error. Trajectories of uninstructed pedestrians in public traffic at a real urban intersection are acquired by a wide angle stereo camera setup in combination with a 3D head tracking framework. Results using this real-world data show that the artificial neural network significantly improves forecast quality compared to other approaches especially for critical traffic scenes including velocity changes such as starting and stopping. For those velocity changes a reduction of position estimation errors of about 21% compared to the Kalman Filter and to extrapolation of polynomials is obtained. By means of a concrete pedestrian-vehicle scenario we demonstrate the benefit of the proposed approach for an advanced driver assistant system in terms of reaction time.
---
paper_title: Modeling cooperative navigation in dense human crowds

paper_content:

For robots to be a part of our daily life, they need to be able to navigate among crowds not only safely but also in a socially compliant fashion. This is a challenging problem because humans tend to navigate by implicitly cooperating with one another to avoid collisions, while heading toward their respective destinations. Previous approaches have used handcrafted functions based on proximity to model human-human and human-robot interactions. However, these approaches can only model simple interactions and fail to generalize for complex crowded settings. In this paper, we develop an approach that models the joint distribution over future trajectories of all interacting agents in the crowd, through a local interaction model that we train using real human trajectory data. The interaction model infers the velocity of each agent based on the spatial orientation of other agents in his vicinity. During prediction, our approach infers the goal of the agent from its past trajectory and uses the learned model to predict its future trajectory. We demonstrate the performance of our method against a state-of-the-art approach on a public dataset and show that our model outperforms when predicting future trajectories for longer horizons.
---
paper_title: Recognition of situation classes at road intersections

paper_content:

The recognition and prediction of situations is an indispensable skill of future driver assistance systems. This study focuses on the recognition of situations involving two vehicles at intersections. For each vehicle, a set of possible future motion trajectories is estimated and rated based on a motion database for a time interval of 2–4 seconds ahead. Realistic situations are generated by a pairwise combination of these individual motion trajectories and classified according to nine categories with a polynomial classifier. In the proposed framework, situations are penalised for which the time to collision significantly exceeds the typical human reaction time. The correspondingly favoured situations are combined by a probabilistic framework, resulting in a more reliable situation recognition and collision detection than obtained based on independent motion hypotheses. The proposed method is evaluated on a real-world differential GPS data set acquired during a test drive of 10 km, including three road intersections. Our method is typically able to recognise the situation correctly about 1–2 seconds before the distance to the intersection centre becomes minimal.
---
paper_title: Estimation of Multivehicle Dynamics by Considering Contextual Information

paper_content:

Human drivers are endowed with an inborn ability to put themselves in the position of other drivers and reason about their behavior and intended actions. State-of-the-art driving-assistance systems, on the other hand, are generally limited to physical models and ad hoc safety rules. In order to drive safely amongst humans, autonomous vehicles need to develop an understanding of the situation in the form of a high-level description of the state of traffic participants. This paper presents a probabilistic model to estimate the state of vehicles by considering interactions between drivers immersed in traffic. The model is defined within a probabilistic filtering framework; estimation and prediction are carried out with statistical inference techniques. Memory requirements increase linearly with the number of vehicles, and thus, it is possible to scale the model to complex scenarios involving many participants. The approach is validated using real-world data collected by a group of interacting ground vehicles.
---
paper_title: Will the Pedestrian Cross? A Study on Pedestrian Path Prediction

paper_content:

Future vehicle systems for active pedestrian safety will not only require a high recognition performance but also an accurate analysis of the developing traffic situation. In this paper, we present a study on pedestrian path prediction and action classification at short subsecond time intervals. We consider four representative approaches: two novel approaches (based on Gaussian process dynamical models and probabilistic hierarchical trajectory matching) that use augmented features derived from dense optical flow and two approaches as baseline that use positional information only (a Kalman filter and its extension to interacting multiple models). In experiments using stereo vision data obtained from a vehicle, we investigate the accuracy of path prediction and action classification at various time horizons, the effect of various errors (image localization, vehicle egomotion estimation), and the benefit of the proposed approaches. The scenario of interest is that of a crossing pedestrian, who might stop or continue walking at the road curbside. Results indicate similar performance of the four approaches on walking motion, with near-linear dynamics. During stopping, however, the two newly proposed approaches, with nonlinear and/or higher order models and augmented motion features, achieve a more accurate position prediction of 10-50 cm at a time horizon of 0-0.77 s around the stopping event.
---
paper_title: Pedestrian Prediction by Planning Using Deep Neural Networks

paper_content:

Accurate traffic participant prediction is the prerequisite for collision avoidance of autonomous vehicles. In this work, we propose to predict pedestrians using goal-directed planning. For this, we infer a mixture density function for possible destinations. We use these destinations as the goal states of a planning stage that performs motion prediction based on common behavior patterns. The patterns are learned by a fully convolutional network operating on maps of the environment. We show that this entire system can be modeled as one monolithic neural network and trained via inverse reinforcement learning. Experimental validation on real world data shows the system's ability to predict both, destinations and trajectories accurately.
---
paper_title: Modelling Smooth Paths Using Gaussian Processes

paper_content:

A generative model based on the gaussian mixture model and gaussian processes is presented in this paper. Typical motion paths are learnt and then used for motion prediction using this model. The principal novel aspect of this approach is the modelling of paths using gaussian processes. It allows the representation of smooth trajectories and avoids discretization problems found in most existing methods. Gaussian processes not only provides a comprehensive and formal theoretical framework to work with, it also lends itself naturally to path clustering using gaussian mixture models. Learning is performed using expectation maximization where the E-Step uses variational methods to maximize its lower bound before optimization over parameters are performed in the M-Step.
---
paper_title: Map-based long term motion prediction for vehicles in traffic environments

paper_content:

Depending on driver intention and current motion state of vehicle, an infinite set of possible future trajectories exists. In this paper we present a stochastic filter which is able to select a representative set of reasonable trajectories from this solution set using additional information from a digital map. This is achieved by representing the map's traffic lanes by their corresponding centerlines. Each detected vehicle is projected on these centerlines, where the necessary information is modeled as a multivariate random variable. With this stochastic model of traffic lanes and the stochastic vector of vehicle detection we define a new stochastic residual vector, which is used both in the lane assignment and during the generation of the motion predictions. For each traffic lane, which is assigned as relevant for a detected vehicle, we generate a new motion hypothesis by using an Extended Kalman Filter. To assess the plausibility of each motion hypothesis we employ an adaptive multivariate MCUSUM algorithm. The implemented stochastic filter is tested by evaluating real data from the Ko-FAS Research Initiative. While traditional motion prediction based on previous and current motion states of a vehicle only provide plausible prediction for a short term time horizon, we show that our approach achieves a reasonable motion hypothesis for long prediction intervals especially in complex scenarios as road intersections.
---
paper_title: Pedestrian Behavior Modeling From Stationary Crowds With Applications to Intelligent Surveillance

paper_content:

Pedestrian behavior modeling and analysis is important for crowd scene understanding and has various applications in video surveillance. Stationary crowd groups are a key factor influencing pedestrian walking patterns but was mostly ignored in the literature. It plays different roles for different pedestrians in a crowded scene and can change over time. In this paper, a novel model is proposed to model pedestrian behaviors by incorporating stationary crowd groups as a key component. Through inference on the interactions between stationary crowd groups and pedestrians, our model can be used to investigate pedestrian behaviors. The effectiveness of the proposed model is demonstrated through multiple applications, including walking path prediction, destination prediction, personality attribute classification, and abnormal event detection. To evaluate our model, two large pedestrian walking route datasets are built. The walking routes of around 15 000 pedestrians from two crowd surveillance videos are manually annotated. The datasets will be released to the public and benefit future research on pedestrian behavior analysis and crowd scene understanding.
---
paper_title: DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents

paper_content:

We introduce a Deep Stochastic IOC RNN Encoder-decoder framework, DESIRE, for the task of future predictions of multiple interacting agents in dynamic scenes. DESIRE effectively predicts future locations of objects in multiple scenes by 1) accounting for the multi-modal nature of the future prediction (i.e., given the same context, future may vary), 2) foreseeing the potential future outcomes and make a strategic prediction based on that, and 3) reasoning not only from the past motion history, but also from the scene context as well as the interactions among the agents. DESIRE achieves these in a single end-to-end trainable neural network model, while being computationally efficient. The model first obtains a diverse set of hypothetical future prediction samples employing a conditional variational auto-encoder, which are ranked and refined by the following RNN scoring-regression module. Samples are scored by accounting for accumulated future rewards, which enables better long-term strategic decisions similar to IOC frameworks. An RNN scene context fusion module jointly captures past motion histories, the semantic scene context and interactions among multiple agents. A feedback mechanism iterates over the ranking and refinement to further boost the prediction accuracy. We evaluate our model on two publicly available datasets: KITTI and Stanford Drone Dataset. Our experiments show that the proposed model significantly improves the prediction accuracy compared to other baseline methods.
---
paper_title: Patch to the Future: Unsupervised Visual Prediction

paper_content:

In this paper we present a conceptually simple but surprisingly powerful method for visual prediction which combines the effectiveness of mid-level visual elements with temporal modeling. Our framework can be learned in a completely unsupervised manner from a large collection of videos. However, more importantly, because our approach models the prediction framework on these mid-level elements, we can not only predict the possible motion in the scene but also predict visual appearances--how are appearances going to change with time. This yields a visual "hallucination" of probable events on top of the scene. We show that our method is able to accurately predict and visualize simple future events, we also show that our approach is comparable to supervised methods for event prediction.
---
paper_title: Camera-based observation of obstacle motions to derive statistical data for mobile robot motion planning

paper_content:

Mobile robots for advanced applications have to act in environments which contain moving obstacles. Motions of obstacles (e.g. humans) usually are not precisely predictable, but neither they are completely random. Long-term observation of obstacle behavior may yield knowledge about prevailing motion patterns. This paper presents concepts and results of an experimental system. Using cameras mounted at the ceiling, the workspace is observed. The image data is processed and transformed into a compact statistical representation of motion patterns. Mobile robot motion planning benefits from such additional knowledge: trajectories are rated in terms of collision probability and expected time for reaching the goal; planning yields efficient paths which are adapted to obstacle behavior.
---
paper_title: Learning Motion Patterns of People for Compliant Robot Motion

paper_content:

Whenever people move through their environments they do not move randomly. Instead, they usually follow specific trajectories or motion patterns corresponding to their intentions. Knowledge about such patterns enables a mobile robot to robustly keep track of persons in its environment and to improve its behavior. In this paper we propose a technique for learning collections of trajectories that characterize typical motion patterns of persons. Data recorded with laser-range finders are clustered using the expectation maximization algorithm. Based on the result of the clustering process, we derive a hidden Markov model that is applied to estimate the current and future positions of persons based on sensory input. We also describe how to incorporate the probabilistic belief about the potential trajectories of persons into the path planning process of a mobile robot. We present several experiments carried out in different environments with a mobile robot equipped with a laser-range scanner and a camera system...
---
paper_title: Deep Learning Driven Visual Path Prediction from a Single Image

paper_content:

Capabilities of inference and prediction are the significant components of visual systems. Visual path prediction is an important and challenging task among them, with the goal to infer the future path of a visual object in a static scene. This task is complicated as it needs high-level semantic understandings of both the scenes and underlying motion patterns in video sequences. In practice, cluttered situations have also raised higher demands on the effectiveness and robustness of models. Motivated by these observations, we propose a deep learning framework, which simultaneously performs deep feature learning for visual representation in conjunction with spatiotemporal context modeling. After that, a unified path-planning scheme is proposed to make accurate path prediction based on the analytic results returned by the deep context models. The highly effective visual representation and deep context models ensure that our framework makes a deep semantic understanding of the scenes and motion patterns, consequently improving the performance on visual path prediction task. In experiments, we extensively evaluate the model’s performance by constructing two large benchmark datasets from the adaptation of video tracking datasets. The qualitative and quantitative experimental results show that our approach outperforms the state-of-the-art approaches and owns a better generalization capability.
---
paper_title: Novel planning-based algorithms for human motion prediction

paper_content:

Human motion prediction from visual tracking is a challenging problem with a wide array of applications such as robotics, video surveillance and situation understanding. Recently, planning-based methods -which assume that people move by planning over a cost function- have emerged as one of the most promising alternatives. Nevertheless, state of the art planning based algorithms have shortcomings regarding their computational complexity and ability to predict for arbitrary time intervals. This paper addresses these shortcomings by leveraging alternative planning techniques (Fast Marching Method) and formulating efficient algorithms for goal estimation and full spatiotemporal prediction with lower complexity than comparable approaches. In preliminary experiments, the proposed method significantly outperforms the accuracy of the current state-of-the-art approach while reducing the computation time by a factor of 30 using a parallel version of our algorithm.
---
paper_title: Predicting actions to act predictably: Cooperative partial motion planning with maximum entropy models

paper_content:

This paper reports on a data-driven motion planning approach for interaction-aware, socially-compliant robot navigation among human agents. Autonomous mobile robots navigating in workspaces shared with human agents require motion planning techniques providing seamless integration and smooth navigation in such. Smooth integration in mixed scenarios calls for two abilities of the robot: predicting actions of others and acting predictably for them. The former requirement requests trainable models of agent behaviors in order to accurately forecast their actions in the future, taking into account their reaction on the robot's decisions. A human-like navigation style of the robot facilitates other agents—most likely not aware of the underlying planning technique applied—to predict the robot motion vice versa, resulting in smoother joint navigation. The approach presented in this paper is based on a feature-based maximum entropy model and is able to guide a robot in an unstructured, real-world environment. The model is trained to predict joint behavior of heterogeneous groups of agents from onboard data of a mobile platform. We evaluate the benefit of interaction-aware motion planning in a realistic public setting with a total distance traveled of over 4 km. Interestingly the motion models learned from human-human interaction did not hold for robot-human interaction, due to the high attention and interest of pedestrians in testing basic braking functionality of the robot.
---
paper_title: Social LSTM: Human Trajectory Prediction in Crowded Spaces

paper_content:

Pedestrians follow different trajectories to avoid obstacles and accommodate fellow pedestrians. Any autonomous vehicle navigating such a scene should be able to foresee the future positions of pedestrians and accordingly adjust its path to avoid collisions. This problem of trajectory prediction can be viewed as a sequence generation task, where we are interested in predicting the future trajectory of people based on their past positions. Following the recent success of Recurrent Neural Network (RNN) models for sequence prediction tasks, we propose an LSTM model which can learn general human movement and predict their future trajectories. This is in contrast to traditional approaches which use hand-crafted functions such as Social forces. We demonstrate the performance of our method on several public datasets. Our model outperforms state-of-the-art methods on some of these datasets. We also analyze the trajectories predicted by our model to demonstrate the motion behaviour learned by our model.
---
paper_title: IMM object tracking for high dynamic driving maneuvers

paper_content:

Classical object tracking approaches use a Kalman-filter with a single dynamic model which is therefore optimised to a single driving maneuver. In contrast the interacting multiple model (IMM) filter allows for several parallel models which are combined to a weighted estimate. Choosing models for different driving modes, such as constant speed, acceleration and strong acceleration changes, the object state estimation can be optimised for highly dynamic driving maneuvers. The paper describes the analysis of Stop&Go situations and the systematic parametrisation of the IMM method based on these statistics. The evaluation of the IMM approach is presented based on real sensor measurements of laser scanners, a radar and a video image processing unit.
---
paper_title: Threat assessment design for driver assistance system at intersections

paper_content:

This paper considers the decision-making problem for a human-driven vehicle crossing a road intersection in the presence of other, potentially errant, drivers. Our approach relies on a novel threat assessment module, which combines an intention predictor based on support vector machines with an efficient threat assessor using rapidly-exploring random trees. This module warns the host driver with the computed threat level and the corresponding best “escape maneuver” through the intersection, if the threat is sufficiently large. Through experimental results with small autonomous and human-driven vehicles, we demonstrate that this threat assessment module can be used in real-time to minimize the risk of collision.
---
paper_title: Socially-aware robot navigation: A learning approach

paper_content:

The ability to act in a socially-aware way is a key skill for robots that share a space with humans. In this paper we address the problem of socially-aware navigation among people that meets objective criteria such as travel time or path length as well as subjective criteria such as social comfort. Opposed to model-based approaches typically taken in related work, we pose the problem as an unsupervised learning problem. We learn a set of dynamic motion prototypes from observations of relative motion behavior of humans found in publicly available surveillance data sets. The learned motion prototypes are then used to compute dynamic cost maps for path planning using an any-angle A* algorithm. In the evaluation we demonstrate that the learned behaviors are better in reproducing human relative motion in both criteria than a Proxemics-based baseline method.
---
paper_title: Activity forecasting

paper_content:

We address the task of inferring the future actions of people from noisy visual input. We denote this task activity forecasting. To achieve accurate activity forecasting, our approach models the effect of the physical environment on the choice of human actions. This is accomplished by the use of state-of-the-art semantic scene understanding combined with ideas from optimal control theory. Our unified model also integrates several other key elements of activity analysis, namely, destination forecasting, sequence smoothing and transfer learning. As proof-of-concept, we focus on the domain of trajectory-based activity analysis from visual input. Experimental results demonstrate that our model accurately predicts distributions over future actions of individuals. We show how the same techniques can improve the results of tracking algorithms by leveraging information about likely goals and trajectories.
---
paper_title: You'll never walk alone: Modeling social behavior for multi-target tracking

paper_content:

Object tracking typically relies on a dynamic model to predict the object's location from its past trajectory. In crowded scenarios a strong dynamic model is particularly important, because more accurate predictions allow for smaller search regions, which greatly simplifies data association. Traditional dynamic models predict the location for each target solely based on its own history, without taking into account the remaining scene objects. Collisions are resolved only when they happen. Such an approach ignores important aspects of human behavior: people are driven by their future destination, take into account their environment, anticipate collisions, and adjust their trajectories at an early stage in order to avoid them. In this work, we introduce a model of dynamic social behavior, inspired by models developed for crowd simulation. The model is trained with videos recorded from birds-eye view at busy locations, and applied as a motion model for multi-people tracking from a vehicle-mounted camera. Experiments on real sequences show that accounting for social interactions and scene knowledge improves tracking performance, especially during occlusions.
---
paper_title: Inferring "Dark Matter" and "Dark Energy" from Videos

paper_content:

This paper presents an approach to localizing functional objects in surveillance videos without domain knowledge about semantic object classes that may appear in the scene. Functional objects do not have discriminative appearance and shape, but they affect behavior of people in the scene. For example, they "attract" people to approach them for satisfying certain needs (e.g., vending machines could quench thirst), or "repel" people to avoid them (e.g., grass lawns). Therefore, functional objects can be viewed as "dark matter", emanating "dark energy" that affects people's trajectories in the video. To detect "dark matter" and infer their "dark energy" field, we extend the Lagrangian mechanics. People are treated as particle-agents with latent intents to approach "dark matter" and thus satisfy their needs, where their motions are subject to a composite "dark energy" field of all functional objects in the scene. We make the assumption that people take globally optimal paths toward the intended "dark matter" while avoiding latent obstacles. A Bayesian framework is used to probabilistically model: people's trajectories and intents, constraint map of the scene, and locations of functional objects. A data-driven Markov Chain Monte Carlo (MCMC) process is used for inference. Our evaluation on videos of public squares and courtyards demonstrates our effectiveness in localizing functional objects and predicting people's trajectories in unobserved parts of the video footage.
---
paper_title: Behavior estimation for a complete framework for human motion prediction in crowded environments

paper_content:

In the present work, we propose and validate a complete probabilistic framework for human motion prediction in urban or social environments. Additionally, we formulate a powerful and useful tool: the human motion behavior estimator. Three different basic behaviors have been detected: Aware, Balanced and Unaware. Our approach is based on the Social Force Model (SFM) and the intentionality prediction BHMIP. The main contribution of the present work is to make use of the behavior estimator for formulating a reliable prediction framework of human trajectories under the influence of dynamic crowds, robots, and in general any moving obstacle. Accordingly, we have demonstrated the great performance of our long-term prediction algorithm, in real scenarios, comparing to other prediction methods.
---
paper_title: Prediction of moving objects in dynamic environments using Kalman filters

paper_content:

In this work, we describe a framework for predicting future positions and orientation of moving obstacles in a time-varying environment using Kalman filtering techniques. No constraint are placed on the obstacles motion. The proposed algorithm can be used in a variety of applications, one of which is robot motion planning in time varying environments. The advantage of using this model compared to reported ones in the literature is it's ability to start the prediction process from the first time step without the need to wait for few time steps before starting the prediction process. Early experimental results are very encouraging.
---
paper_title: Conditional transition maps: Learning motion patterns in dynamic environments

paper_content:

In this paper we introduce a method for learning motion patterns in dynamic environments. Representations of dynamic environments have recently received an increasing amount of attention in the research community. Understanding dynamic environments is seen as one of the key challenges in order to enable autonomous navigation in real-world scenarios. However, representing the temporal dimension is a challenge yet to be solved. In this paper we introduce a spatial representation, which encapsulates the statistical dynamic behavior observed in the environment. The proposed Conditional Transition Map (CTMap) is a grid-based representation that associates a probability distribution for an object exiting the cell, given its entry direction. The transition parameters are learned from a temporal signal of occupancy on cells by using a local-neighborhood cross-correlation method. In this paper, we introduce the CTMap, the learning approach and present a proof-of-concept method for estimating future paths of dynamic objects, called Conditional Probability Propagation Tree (CPPTree). The evaluation is done using a real-world dataset collected at a busy roundabout.
---
paper_title: People tracking with human motion predictions from social forces

paper_content:

For many tasks in populated environments, robots need to keep track of current and future motion states of people. Most approaches to people tracking make weak assumptions on human motion such as constant velocity or acceleration. But even over a short period, human behavior is more complex and influenced by factors such as the intended goal, other people, objects in the environment, and social rules. This motivates the use of more sophisticated motion models for people tracking especially since humans frequently undergo lengthy occlusion events. In this paper, we consider computational models developed in the cognitive and social science communities that describe individual and collective pedestrian dynamics for tasks such as crowd behavior analysis. In particular, we integrate a model based on a social force concept into a multi-hypothesis target tracker. We show how the refined motion predictions translate into more informed probability distributions over hypotheses and finally into a more robust tracking behavior and better occlusion handling. In experiments in indoor and outdoor environments with data from a laser range finder, the social force model leads to more accurate tracking with up to two times fewer data association errors.
---
paper_title: Unfreezing the robot: Navigation in dense, interacting crowds

paper_content:

In this paper, we study the safe navigation of a mobile robot through crowds of dynamic agents with uncertain trajectories. Existing algorithms suffer from the “freezing robot” problem: once the environment surpasses a certain level of complexity, the planner decides that all forward paths are unsafe, and the robot freezes in place (or performs unnecessary maneuvers) to avoid collisions. Since a feasible path typically exists, this behavior is suboptimal. Existing approaches have focused on reducing the predictive uncertainty for individual agents by employing more informed models or heuristically limiting the predictive covariance to prevent this overcautious behavior. In this work, we demonstrate that both the individual prediction and the predictive uncertainty have little to do with the frozen robot problem. Our key insight is that dynamic agents solve the frozen robot problem by engaging in “joint collision avoidance”: They cooperatively make room to create feasible trajectories. We develop IGP, a nonparametric statistical model based on dependent output Gaussian processes that can estimate crowd interaction from data. Our model naturally captures the non-Markov nature of agent trajectories, as well as their goal-driven navigation. We then show how planning in this model can be efficiently implemented using particle based inference. Lastly, we evaluate our model on a dataset of pedestrians entering and leaving a building, first comparing the model with actual pedestrians, and find that the algorithm either outperforms human pedestrians or performs very similarly to the pedestrians. We also present an experiment where a covariance reduction method results in highly overcautious behavior, while our model performs desirably.
---
paper_title: Generating Long-term Trajectories Using Deep Hierarchical Networks

paper_content:

We study the problem of modeling spatiotemporal trajectories over long time horizons using expert demonstrations. For instance, in sports, agents often choose action sequences with long-term goals in mind, such as achieving a certain strategic position. Conventional policy learning approaches, such as those based on Markov decision processes, generally fail at learning cohesive long-term behavior in such high-dimensional state spaces, and are only effective when fairly myopic decision-making yields the desired behavior. The key difficulty is that conventional models are ``single-scale'' and only learn a single state-action policy. We instead propose a hierarchical policy class that automatically reasons about both long-term and short-term goals, which we instantiate as a hierarchical neural network. We showcase our approach in a case study on learning to imitate demonstrated basketball trajectories, and show that it generates significantly more realistic trajectories compared to non-hierarchical baselines as judged by professional sports analysts.
---
paper_title: Understanding interactions between traffic participants based on learned behaviors

paper_content:

Predicting vehicles' behaviors in a traffic scene can be very challenging due to many influences. Especially interactions with other traffic participants like vehicles or pedestrians are very crucial for the future movement while they are hard to model even with expert knowledge.
---
paper_title: Knowledge Transfer for Scene-specific Motion Prediction

paper_content:

When given a single frame of the video, humans can not only interpret the content of the scene, but also they are able to forecast the near future. This ability is mostly driven by their rich prior knowledge about the visual world, both in terms of (i) the dynamics of moving agents, as well as (ii) the semantic of the scene. In this work we exploit the interplay between these two key elements to predict scene-specific motion patterns. First, we extract patch descriptors encoding the probability of moving to the adjacent patches, and the probability of being in that particular patch or changing behavior. Then, we introduce a Dynamic Bayesian Network which exploits this scene specific knowledge for trajectory prediction. Experimental results demonstrate that our method is able to accurately predict trajectories and transfer predictions to a novel scene characterized by similar elements.
---
paper_title: Incremental learning of human social behaviors with feature-based spatial effects

paper_content:

In the near future, robots will exist in our home, office, hospital, school and all other public places. In other words, robots have to work and share environments with human. However, there exist many invisible social rules or social protocols in human society. For example, people usually stand in a line to access certain services. It seems that there is a social force or a “spatial effect” which forces people to perform the behavior of standing in a line. Unfortunately, these spatial effects are usually invisible and immeasurable. It makes social behavior understanding to be a difficult problem. The robot that does not comprehend these spatial effects might harm people or itself. Previous methods of behavior understanding mainly only targeted certain scenarios or environments. This paper follows the framework of SBCM (Spatial Behavior Cognition Model)[3] and extends to feature-based spatial effects. By utilizing feature to represent the spatial effect, it is easier to apply known spatial effects to different environments. This paper also demonstrates that the robot is able to learn and execute some common social behaviors under the proposed method. Several simulations and experiments are conducted to verify the proposed idea in this paper.
---
paper_title: Context-Based Path Prediction for Targets with Switching Dynamics

paper_content:

Anticipating future situations from streaming sensor data is a key perception challenge for mobile robotics and automated vehicles. We address the problem of predicting the path of objects with multiple dynamic modes. The dynamics of such targets can be described by a Switching Linear Dynamical System (SLDS). However, predictions from this probabilistic model cannot anticipate when a change in dynamic mode will occur. We propose to extract various types of cues with computer vision to provide context on the target’s behavior, and incorporate these in a Dynamic Bayesian Network (DBN). The DBN extends the SLDS by conditioning the mode transition probabilities on additional context states. We describe efficient online inference in this DBN for probabilistic path prediction, accounting for uncertainty in both measurements and target behavior. Our approach is illustrated on two scenarios in the Intelligent Vehicles domain concerning pedestrians and cyclists, so-called Vulnerable Road Users (VRUs). Here, context cues include the static environment of the VRU, its dynamic environment, and its observed actions. Experiments using stereo vision data from a moving vehicle demonstrate that the proposed approach results in more accurate path prediction than SLDS at the relevant short time horizon (1 s). It slightly outperforms a computationally more demanding state-of-the-art method.
---
paper_title: Pedestrian Path Prediction with Recursive Bayesian Filters: A Comparative Study

paper_content:

In the context of intelligent vehicles, we perform a comparative study on recursive Bayesian filters for pedestrian path prediction at short time horizons (< 2s). We consider Extended Kalman Filters (EKF) based on single dynamical models and Interacting Multiple Models (IMM) combining several such basic models (constant velocity/acceleration/turn). These are applied to four typical pedestrian motion types (crossing, stopping, bending in, starting). Position measurements are provided by an external state-of-the-art stereo vision-based pedestrian detector. We investigate the accuracy of position estimation and path prediction, and the benefit of the IMMs vs. the simpler single dynamical models. Special care is given to the proper sensor modeling and parameter optimization. The dataset and evaluation framework are made public to facilitate benchmarking.
---
paper_title: Voronoi tracking: location estimation using sparse and noisy sensor data

paper_content:

Tracking the activity of people in indoor environments has gained considerable attention in the robotics community over the last years. Most of the existing approaches are based on sensors, which allow to accurately determining the locations of people but do not provide means to distinguish between different persons. In this paper we propose a novel approach to tracking moving objects and their identity using noisy, sparse information collected by id-sensors such as infrared and ultrasound badge systems. The key idea of our approach is to use particle filters to estimate the locations of people on the Voronoi graph of the environment. By restricting particles to a graph, we make use of the inherent structure of indoor environments. The approach has two key advantages. First, it is by far more efficient and robust than unconstrained particle filters. Second, the Voronoi graph provides a natural discretization of human motion, which allows us to apply unsupervised learning techniques to derive typical motion patterns of the people in the environment. Experiments using a robot to collect ground-truth data indicate the superior performance of Voronoi tracking. Furthermore, we demonstrate that EM-based learning of behavior patterns increases the tracking performance and provides valuable information for high-level behavior recognition.
---
paper_title: Motion prediction of moving objects based on autoregressive model

paper_content:

In this paper, we describe a framework for predicting future positions and orientation of moving obstacles in a time-varying environment using autoregressive model (ARM) with conditional maximum likelihood estimate of the model parameters. No constraints are placed on the obstacles motion. The proposed algorithm can be used in a variety of applications, one of which is robot motion planning in time varying environments.
---
paper_title: Planning-based prediction for pedestrians

paper_content:

We present a novel approach for determining robot movements that efficiently accomplish the robot's tasks while not hindering the movements of people within the environment. Our approach models the goal-directed trajectories of pedestrians using maximum entropy inverse optimal control. The advantage of this modeling approach is the generality of its learned cost function to changes in the environment and to entirely different environments. We employ the predictions of this model of pedestrian trajectories in a novel incremental planner and quantitatively show the improvement in hindrance-sensitive robot trajectory planning provided by our approach.
---
paper_title: A mobile robot that understands pedestrian spatial behaviors

paper_content:

In human society, there are many invisible social rules or spatial effects existing in our environments. The robot that does not comprehend these spatial effects might harm people or itself. This paper presents a spatial behavior cognition model (SBCM) to describe the spatial effects existing between people and people, people and environments. By understanding the spatial effects in human-lived environments, the robot not only predicts pedestrian intentions and trajectories but also behaves socially acceptable motions. Moreover, the concept of pedestrian ego-graph (PEG) is proposed to efficiently query pedestrian-like paths for trajectory prediction. Model evaluation and experiments are shown to verify the proposed idea in this paper.
---
paper_title: Using EM to learn motion behaviors of persons with mobile robots

paper_content:

We propose a method for learning models of people's motion behaviors in indoor environments. As people move through their environments, they do not move randomly. Instead, they often engage in typical motion patterns, related to specific locations that they might be interested in approaching and specific trajectories that they might follow in doing so. Knowledge about such patterns may enable a mobile robot to develop improved people following and obstacle avoidance skills. This paper proposes an algorithm that learns collections of typical trajectories that characterize a person's motion patterns. Data, recorded by mobile robots equipped with laser-range finders, is clustered into different types of motion using the popular expectation maximization algorithm, while simultaneously learning multiple motion patterns. Experimental results, obtained using data collected in a domestic residence and in an office building, illustrate that highly predictive models of human motion patterns can be learned.
---
paper_title: Enabling Flow Awareness for Mobile Robots in Partially Observable Environments

paper_content:

Understanding the environment is a key requirement for any autonomous robot operation. There is extensive research on mapping geometric structure and perceiving objects. However, the environment is ...
---
paper_title: Pedestrian Group Behavior in a Cellular Automaton

paper_content:

The study of crowd movement is very important for planning mass events and evacuations. Although many potentially critical incidents are hardly predictable, others like clogging, can be anticipated. Studies have shown that pedestrians in social groups frequently contribute the biggest part to crowds, and that these groups have a significant impact on crowd movement. Furthermore social cooperative behaviour does not stop in emergency situations, but continues or even gets stronger. We employ a cellular automaton with attractive and repulsive potentials for the simulation of individual pedestrians. Then we explain how to integrate a concise model for social groups suitable for the basic modelling techniques. Further aspects of group behavior are discussed, namely large groups and the behavior after a group separation, and we propose a model for these. These investigations lead us to the conclusion that there is a growing need of agent based modelling when facing advanced aspects of pedestrian crowd behavior.
---
paper_title: Real-Time Predictive Modeling and Robust Avoidance of Pedestrians with Uncertain, Changing Intentions

paper_content:

To plan safe trajectories in urban environments, autonomous vehicles must be able to quickly assess the future intentions of dynamic agents. Pedestrians are particularly challenging to model, as their motion patterns are often uncertain and/or unknown a priori. This paper presents a novel changepoint detection and clustering algorithm that, when coupled with offline unsupervised learning of a Gaussian process mixture model (DPGP), enables quick detection of changes in intent and online learning of motion patterns not seen in prior training data. The resulting long-term movement predictions demonstrate improved accuracy relative to offline learning alone, in terms of both intent and trajectory prediction. By embedding these predictions within a chance-constrained motion planner, trajectories which are probabilistically safe to pedestrian motions can be identified in real-time. Hardware experiments demonstrate that this approach can accurately predict pedestrian motion patterns from onboard sensor/perception data and facilitate robust navigation within a dynamic environment.
---
paper_title: Human Motion Prediction Under Social Grouping Constraints

paper_content:

Accurate long-term prediction of human motion in populated spaces is an important but difficult task for mobile robots and intelligent vehicles. What makes this task challenging is that human motion is influenced by a large variety of factors including the person's intention, the presence, attributes, actions, social relations and social norms of other surrounding agents, and the geometry and semantics of the environment. In this paper, we consider the problem of computing human motion predictions that account for such factors. We formulate the task as an MDP planning problem with stochastic policies and propose a weighted random walk algorithm in which each agent is locally influenced by social forces from other nearby agents. The novelty of this paper is that we incorporate social grouping information into the prediction process reflecting the soft formation constraints that groups typically impose to their members' motion. We show that our method makes more accurate predictions than three state-of-the-art methods in terms of probabilistic and geometrical performance metrics.
---
paper_title: Real-Time Certified Probabilistic Pedestrian Forecasting

paper_content:

The success of autonomous systems will depend upon their ability to safely navigate human-centric environments. This motivates the need for a real-time, probabilistic forecasting algorithm for pedestrians, cyclists, and other agents since these predictions will form a necessary step in assessing the risk of any action. This letter presents a novel approach to probabilistic forecasting for pedestrians based on weighted sums of ordinary differential equations that are learned from historical trajectory information within a fixed scene. The resulting algorithm is embarrassingly parallel and is able to work at real-time speeds using a naive Python implementation. The quality of predicted locations of agents generated by the proposed algorithm is validated on a variety of examples and is considerably higher than existing state of the art approaches over long time horizons.
---
paper_title: Improving data association by joint modeling of pedestrian trajectories and groupings

paper_content:

We consider the problem of data association in a multiperson tracking context. In semi-crowded environments, people are still discernible as individually moving entities, that undergo many interactions with other people in their direct surrounding. Finding the correct association is therefore difficult, but higher-order social factors, such as group membership, are expected to ease the problem. However, estimating group membership is a chicken-and-egg problem: knowing pedestrian trajectories, it is rather easy to find out possible groupings in the data, but in crowded scenes, it is often difficult to estimate closely interacting trajectories without further knowledge about groups. To this end, we propose a third-order graphical model that is able to jointly estimate correct trajectories and group memberships over a short time window. A set of experiments on challenging data underline the importance of joint reasoning for data association in crowded scenarios.
---
paper_title: Using road topology to improve cyclist path prediction

paper_content:

We learn motion models for cyclist path prediction on real-world tracks obtained from a moving vehicle, and propose to exploit the local road topology to obtain better predictive distributions. The tracks are extracted from the Tsinghua-Daimler Cyclist Benchmark for cyclist detection, and corrected for vehicle egomotion. Tracks are then spatially aligned to local curves and crossings in the road. We study a standard approach for path prediction in the literature based on Kalman Filters, as well as a mixture of specialized filters related to specific road orientations at junctions. Our experiments demonstrate an improved prediction accuracy (up to 20% on sharp turns) of mixing specialized motion models for canonical directions, and prior knowledge on the road topology. The new track data complements the existing video, disparity and annotation data of the original benchmark, and will be made publicly available.
---
paper_title: Modelling pedestrian trajectory patterns with Gaussian processes

paper_content:

We propose a non-parametric model for pedestrian motion based on Gaussian Process regression, in which trajectory data are modelled by regressing relative motion against current position. We show how the underlying model can be learned in an unsupervised fashion, demonstrating this on two databases collected from static surveillance cameras. We furthermore exemplify the use of model for prediction, comparing the recently proposed GP-Bayesfilters with a Monte Carlo method. We illustrate the benefit of this approach for long term motion prediction where parametric models such as Kalman Filters would perform poorly.
---
paper_title: Stochastic reachable sets of interacting traffic participants

paper_content:

Knowledge about the future development of a certain road traffic situation is indispensable for safe path planning of autonomous ground vehicles or action selection of intelligent driver assistance systems. Due to a significant uncertainty about the future behavior of traffic participants, the prediction of traffic situations should be computed in a probabilistic setting. Under consideration of the dynamics of traffic participants, their future position is computed probabilistically by Markov chains that are obtained with methods known from hybrid verification. The characteristic feature of the presented approach is that all possible behaviors of traffic participants are considered, allowing to identify any dangerous future situation. The novel contribution of this work is the explicit modeling of the interaction of traffic participants, which leads to a more accurate prediction of their positions. Results are demonstrated for different traffic situations.
---
paper_title: Intent-aware long-term prediction of pedestrian motion

paper_content:

We present a method to predict long-term motion of pedestrians, modeling their behavior as jump-Markov processes with their goal a hidden variable. Assuming approximately rational behavior, and incorporating environmental constraints and biases, including time-varying ones imposed by traffic lights, we model intent as a policy in a Markov decision process framework. We infer pedestrian state using a Rao-Blackwellized filter, and intent by planning according to a stochastic policy, reflecting individual preferences in aiming at the same goal.
---
paper_title: Learning intentions for improved human motion prediction

paper_content:

For many tasks, robots need to operate in human populated environments. Human motion prediction therefore is gaining importance. The concept of social forces defines virtual repelling and attracting forces from and to obstacles and points of interest. These social forces can then be used to model typical human movements given an environment and a person's intention. This work shows how such models can exploit typical motion patterns summarized by growing hidden Markov models (GHMMs) that can be learned from data online and without human intervention. An extensive series of experiments shows that exploiting the intended position estimated using a GHMM within a social forces based motion model yields a significant performance gain in comparison with the standard constant velocity-based models.
---
paper_title: Modelling subgroup behaviour in crowd dynamics DEM simulation

paper_content:

This paper investigates the behaviour of subgroups in crowd dynamics by means of filming and observation. An existing crowd modelling program, CrowdDMX, based on a discrete element model (DEM) has been modified on the basis of observations made in this paper and literature. Each person is represented as three overlapping circles and motion is modelled in a Newtonian manner. It incorporates psychological forces as well as physical forces in a 2D time-stepping environment. The DEM model was modified to include realistic subgroup behaviour, representing people in the crowd desiring to stay together (families, friends, etc.). Subgroup psychological forces were incorporated. The previous model only simulated individuals moving independently, which was unrealistic in some situations as shown by the observation and filming part of the study. The revised program models subgroups realistically including the tendency to avoid subgroup division in cases of contra-flow.
---
paper_title: Forecasting Interactive Dynamics of Pedestrians with Fictitious Play

paper_content:

We develop predictive models of pedestrian dynamics by encoding the coupled nature of multi-pedestrian interaction using game theory, and deep learning-based visual analysis to estimate person-specific behavior parameters. Building predictive models for multi-pedestrian interactions however, is very challenging due to two reasons: (1) the dynamics of interaction are complex interdependent processes, where the predicted behavior of one pedestrian can affect the actions taken by others and (2) dynamics are variable depending on an individuals physical characteristics (e.g., an older person may walk slowly while the younger person may walk faster). To address these challenges, we (1) utilize concepts from game theory to model the interdependent decision making process of multiple pedestrians and (2) use visual classifiers to learn a mapping from pedestrian appearance to behavior parameters. We evaluate our proposed model on several public multiple pedestrian interaction video datasets. Results show that our strategic planning model explains human interactions 25% better when compared to state-of-the-art methods.
---
paper_title: Who are you with and where are you going?

paper_content:

We propose an agent-based behavioral model of pedestrians to improve tracking performance in realistic scenarios. In this model, we view pedestrians as decision-making agents who consider a plethora of personal, social, and environmental factors to decide where to go next. We formulate prediction of pedestrian behavior as an energy minimization on this model. Two of our main contributions are simple, yet effective estimates of pedestrian destination and social relationships (groups). Our final contribution is to incorporate these hidden properties into an energy formulation that results in accurate behavioral prediction. We evaluate both our estimates of destination and grouping, as well as our accuracy at prediction and tracking against state of the art behavioral model and show improvements, especially in the challenging observational situation of infrequent appearance observations–something that might occur in thousands of webcams available on the Internet.
---
paper_title: Long-term Path Prediction in Urban Scenarios using Circular Distributions

paper_content:

Abstract Human ability to foresee the near future plays a key role in everyone's life to prevent potentially dangerous situations. To be able to make predictions is crucial when people have to interact with the surrounding environment. Modeling such capability can lead to the design of automated warning systems and provide moving robots with an intelligent way of interaction with changing situation. In this work we focus on a typical urban human-scene where we aim at predicting an agent's behavior using a stochastic model. In this approach, we fuse the various factors that would contribute to a human motion in different contexts. Our method uses previously observed trajectories to build point-wise circular distributions that after combination, provide a statistical smooth prediction towards the most likely areas. More specifically, a ray-launching procedure, based on a semantic segmentation, gives a coarse scene representation for collision avoidance; a nearly-constant velocity dynamic model smooths the acceleration progression and knowledge of the agent's destination may further steer the path prediction. Experimental results in structured scenes, validate the effectiveness of the method in predicting paths in comparison to actual trajectories.
---
paper_title: Pedestrian's Trajectory Forecast in Public Traffic with Artificial Neural Networks

paper_content:

This paper focuses on forecasting of pedestrian's short-time trajectories up to 2.5 s for traffic safety applications. We present a self-learning approach based on artificial neural network movement models and compare it to traditional constant velocity Kalman Filter prediction and extrapolation of polynomials fitted using a least-squares error. Trajectories of uninstructed pedestrians in public traffic at a real urban intersection are acquired by a wide angle stereo camera setup in combination with a 3D head tracking framework. Results using this real-world data show that the artificial neural network significantly improves forecast quality compared to other approaches especially for critical traffic scenes including velocity changes such as starting and stopping. For those velocity changes a reduction of position estimation errors of about 21% compared to the Kalman Filter and to extrapolation of polynomials is obtained. By means of a concrete pedestrian-vehicle scenario we demonstrate the benefit of the proposed approach for an advanced driver assistant system in terms of reaction time.
---
paper_title: Learning to navigate through crowded environments

paper_content:

The goal of this research is to enable mobile robots to navigate through crowded environments such as indoor shopping malls, airports, or downtown side walks. The key research question addressed in this paper is how to learn planners that generate human-like motion behavior. Our approach uses inverse reinforcement learning (IRL) to learn human-like navigation behavior based on example paths. Since robots have only limited sensing, we extend existing IRL methods to the case of partially observable environments. We demonstrate the capabilities of our approach using a realistic crowd flow simulator in which we modeled multiple scenarios in crowded environments. We show that our planner learned to guide the robot along the flow of people when the environment is crowded, and along the shortest path if no people are around.
---
paper_title: Modeling cooperative navigation in dense human crowds

paper_content:

For robots to be a part of our daily life, they need to be able to navigate among crowds not only safely but also in a socially compliant fashion. This is a challenging problem because humans tend to navigate by implicitly cooperating with one another to avoid collisions, while heading toward their respective destinations. Previous approaches have used handcrafted functions based on proximity to model human-human and human-robot interactions. However, these approaches can only model simple interactions and fail to generalize for complex crowded settings. In this paper, we develop an approach that models the joint distribution over future trajectories of all interacting agents in the crowd, through a local interaction model that we train using real human trajectory data. The interaction model infers the velocity of each agent based on the spatial orientation of other agents in his vicinity. During prediction, our approach infers the goal of the agent from its past trajectory and uses the learned model to predict its future trajectory. We demonstrate the performance of our method against a state-of-the-art approach on a public dataset and show that our model outperforms when predicting future trajectories for longer horizons.
---
paper_title: Pedestrian Prediction by Planning Using Deep Neural Networks

paper_content:

Accurate traffic participant prediction is the prerequisite for collision avoidance of autonomous vehicles. In this work, we propose to predict pedestrians using goal-directed planning. For this, we infer a mixture density function for possible destinations. We use these destinations as the goal states of a planning stage that performs motion prediction based on common behavior patterns. The patterns are learned by a fully convolutional network operating on maps of the environment. We show that this entire system can be modeled as one monolithic neural network and trained via inverse reinforcement learning. Experimental validation on real world data shows the system's ability to predict both, destinations and trajectories accurately.
---
paper_title: Simulating and Evaluating the Local Behavior of Small Pedestrian Groups

paper_content:

Recent advancements in local methods have significantly improved the collision avoidance behavior of virtual characters. However, existing methods fail to take into account that in real life pedestrians tend to walk in small groups, consisting mainly of pairs or triples of individuals. We present a novel approach to simulate the walking behavior of such small groups. Our model describes how group members interact with each other, with other groups and individuals. We highlight the potential of our method through a wide range of test-case scenarios. We evaluate the results from our simulations using a number of quantitative quality metrics, and also provide visual and numerical comparisons with video footages of real crowds.
---
paper_title: GLMP- realtime pedestrian path prediction using global and local movement patterns

paper_content:

We present a novel real-time algorithm to predict the path of pedestrians in cluttered environments. Our approach makes no assumption about pedestrian motion or crowd density, and is useful for short-term as well as long-term prediction. We interactively learn the characteristics of pedestrian motion and movement patterns from 2D trajectories using Bayesian inference. These include local movement patterns corresponding to the current and preferred velocities and global characteristics such as entry points and movement features. Our approach involves no precomputation and we demonstrate the real-time performance of our prediction algorithm on sparse and noisy trajectory data extracted from dense indoor and outdoor crowd videos. The combination of local and global movement patterns can improve the accuracy of long-term prediction by 12–18% over prior methods in high-density videos.
---
paper_title: DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents

paper_content:

We introduce a Deep Stochastic IOC RNN Encoder-decoder framework, DESIRE, for the task of future predictions of multiple interacting agents in dynamic scenes. DESIRE effectively predicts future locations of objects in multiple scenes by 1) accounting for the multi-modal nature of the future prediction (i.e., given the same context, future may vary), 2) foreseeing the potential future outcomes and make a strategic prediction based on that, and 3) reasoning not only from the past motion history, but also from the scene context as well as the interactions among the agents. DESIRE achieves these in a single end-to-end trainable neural network model, while being computationally efficient. The model first obtains a diverse set of hypothetical future prediction samples employing a conditional variational auto-encoder, which are ranked and refined by the following RNN scoring-regression module. Samples are scored by accounting for accumulated future rewards, which enables better long-term strategic decisions similar to IOC frameworks. An RNN scene context fusion module jointly captures past motion histories, the semantic scene context and interactions among multiple agents. A feedback mechanism iterates over the ranking and refinement to further boost the prediction accuracy. We evaluate our model on two publicly available datasets: KITTI and Stanford Drone Dataset. Our experiments show that the proposed model significantly improves the prediction accuracy compared to other baseline methods.
---
paper_title: Human Motion Behaviour Aware Planner (HMBAP) for path planning in dynamic human environments

paper_content:

For a robot navigating in a human inhabited dynamic environment, the knowledge of how the robot's movement can influence the trajectory of people around it can be very valuable. In this work we present a Human Motion Behaviour Aware Planner (HMBAP) which incorporates a Human Motion Behaviour Model (HMBM) in its planning stage to take advantage of this. HMBM is an obstacle avoidance model for people based on social forces which gives the robot an understanding of how people would react to its planned path. This information is useful for the robot to avoid imminent collisions with people in constricted spaces. The resulting robot behaviour is similar to how a human would move (in terms of avoidance behaviour) in a similar situation. We believe that this is a desirable feature for a robot navigating in a human inhabited environment. Our method shows good human-like navigation behaviour in situations where past methods fail to find a solution.
---
paper_title: Building a human behavior map from local observations

paper_content:

This paper presents a novel method for classifying regions from human movements in service robots' working environments. The entire space is segmented subject to the class type according to the functionality or affordance of each place which accommodates a typical human behavior. This is achieved based on a grid map in two steps. First a probabilistic model is developed to capture human movements for each grid cell by using a non-ergodic HMM. Then the learned transition probabilities corresponding to these movements are used to cluster all cells by using the K-means algorithm. The knowledge of typical human movements for each location, represented by the prototypes from K-means and summarized in a ‘behavior-based map’, enables a robot to adjust the strategy of interacting with people according to where they are located, and thus greatly enhances its capability to assist people. The performance of the proposed classification method is demonstrated by experimental results from 8 hours of data that are collected in a kitchen environment.
---
paper_title: Probabilistic Autonomous Robot Navigation in Dynamic Environments with Human Motion Prediction

paper_content:

This paper considers the problem of autonomous robot navigation in dynamic and congested environments. The predictive navigation paradigm is proposed where probabilistic planning is integrated with obstacle avoidance along with future motion prediction of humans and/or other obstacles. Predictive navigation is performed in a global manner with the use of a hierarchical Partially Observable Markov Decision Process (POMDP) that can be solved on-line at each time step and provides the actual actions the robot performs. Obstacle avoidance is performed within the predictive navigation model with a novel approach by deciding paths to the goal position that are not obstructed by other moving objects movement with the use of future motion prediction and by enabling the robot to increase or decrease its speed of movement or by performing detours. The robot is able to decide which obstacle avoidance behavior is optimal in each case within the unified navigation model employed.
---
paper_title: Human-robot co-navigation using anticipatory indicators of human walking motion

paper_content:

Mobile, interactive robots that operate in human-centric environments need the capability to safely and efficiently navigate around humans. This requires the ability to sense and predict human motion trajectories and to plan around them. In this paper, we present a study that supports the existence of statistically significant biomechanical turn indicators of human walking motions. Further, we demonstrate the effectiveness of these turn indicators as features in the prediction of human motion trajectories. Human motion capture data is collected with predefined goals to train and test a prediction algorithm. Use of anticipatory features results in improved performance of the prediction algorithm. Lastly, we demonstrate the closed-loop performance of the prediction algorithm using an existing algorithm for motion planning within dynamic environments. The anticipatory indicators of human walking motion can be used with different prediction and/or planning algorithms for robotics; the chosen planning and prediction algorithm demonstrates one such implementation for human-robot co-navigation.
---
paper_title: Camera-based observation of obstacle motions to derive statistical data for mobile robot motion planning

paper_content:

Mobile robots for advanced applications have to act in environments which contain moving obstacles. Motions of obstacles (e.g. humans) usually are not precisely predictable, but neither they are completely random. Long-term observation of obstacle behavior may yield knowledge about prevailing motion patterns. This paper presents concepts and results of an experimental system. Using cameras mounted at the ceiling, the workspace is observed. The image data is processed and transformed into a compact statistical representation of motion patterns. Mobile robot motion planning benefits from such additional knowledge: trajectories are rated in terms of collision probability and expected time for reaching the goal; planning yields efficient paths which are adapted to obstacle behavior.
---
paper_title: Learning Social Etiquette: Human Trajectory Understanding In Crowded Scenes

paper_content:

Humans navigate crowded spaces such as a university campus by following common sense rules based on social etiquette. In this paper, we argue that in order to enable the design of new target tracking or trajectory forecasting methods that can take full advantage of these rules, we need to have access to better data in the first place. To that end, we contribute a new large-scale dataset that collects videos of various types of targets (not just pedestrians, but also bikers, skateboarders, cars, buses, golf carts) that navigate in a real world outdoor environment such as a university campus. Moreover, we introduce a new characterization that describes the “social sensitivity” at which two targets interact. We use this characterization to define “navigation styles” and improve both forecasting models and state-of-the-art multi-target tracking–whereby the learnt forecasting models help the data association step.
---
paper_title: Pedestrian Path, Pose, and Intention Prediction Through Gaussian Process Dynamical Models and Pedestrian Activity Recognition

paper_content:

According to several reports published by worldwide organizations, thousands of pedestrians die in road accidents every year. Due to this fact, vehicular technologies have been evolving with the intent of reducing these fatalities. This evolution has not finished yet, since, for instance, the predictions of pedestrian paths could improve the current automatic emergency braking systems. For this reason, this paper proposes a method to predict future pedestrian paths, poses, and intentions up to 1 s in advance. This method is based on balanced Gaussian process dynamical models (B-GPDMs), which reduce the 3-D time-related information extracted from key points or joints placed along pedestrian bodies into low-dimensional spaces. The B-GPDM is also capable of inferring future latent positions and reconstruct their associated observations. However, learning a generic model for all kinds of pedestrian activities normally provides less accurate predictions. For this reason, the proposed method obtains multiple models of four types of activity, i.e., walking, stopping, starting, and standing, and selects the most similar model to estimate future pedestrian states. This method detects starting activities 125 ms after the gait initiation with an accuracy of 80% and recognizes stopping intentions 58.33 ms before the event with an accuracy of 70%. Concerning the path prediction, the mean error for stopping activities at a time-to-event (TTE) of 1 s is 238.01 ± 206.93 mm and, for starting actions, the mean error at a TTE of 0 s is 331.93 ± 254.73 mm.
---
paper_title: Learning Motion Patterns of People for Compliant Robot Motion

paper_content:

Whenever people move through their environments they do not move randomly. Instead, they usually follow specific trajectories or motion patterns corresponding to their intentions. Knowledge about such patterns enables a mobile robot to robustly keep track of persons in its environment and to improve its behavior. In this paper we propose a technique for learning collections of trajectories that characterize typical motion patterns of persons. Data recorded with laser-range finders are clustered using the expectation maximization algorithm. Based on the result of the clustering process, we derive a hidden Markov model that is applied to estimate the current and future positions of persons based on sensory input. We also describe how to incorporate the probabilistic belief about the potential trajectories of persons into the path planning process of a mobile robot. We present several experiments carried out in different environments with a mobile robot equipped with a laser-range scanner and a camera system...
---
paper_title: Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning

paper_content:

Finding feasible, collision-free paths for multiagent systems can be challenging, particularly in non-communicating scenarios where each agent's intent (e.g. goal) is unobservable to the others. In particular, finding time efficient paths often requires anticipating interaction with neighboring agents, the process of which can be computationally prohibitive. This work presents a decentralized multiagent collision avoidance algorithm based on a novel application of deep reinforcement learning, which effectively offloads the online computation (for predicting interaction patterns) to an offline learning procedure. Specifically, the proposed approach develops a value network that encodes the estimated time to the goal given an agent's joint configuration (positions and velocities) with its neighbors. Use of the value network not only admits efficient (i.e., real-time implementable) queries for finding a collision-free velocity vector, but also considers the uncertainty in the other agents' motion. Simulation results show more than 26 percent improvement in paths quality (i.e., time to reach the goal) when compared with optimal reciprocal collision avoidance (ORCA), a state-of-the-art collision avoidance strategy.
---
paper_title: Intentional motion on-line learning and prediction

paper_content:

Predicting motion of humans, animals and other objects which move according to internal plans is a challenging problem. Most existing approaches operate in two stages: (a) learning typical motion patterns by observing an environment and (b) predicting future motion on the basis of the learned patterns. In existing techniques, learning is performed off-line, hence, it is impossible to refine the existing knowledge on the basis of the new observations obtained during the prediction phase. We propose an approach which uses hidden Markov models (HMMs) to represent motion patterns. It is different from similar approaches because it is able to learn and predict in a concurrent fashion thanks to a novel approximate learning approach, based on the growing neural gas algorithm, which estimates both HMM parameters and structure. The found structure has the property of being a planar graph, thus enabling exact inference in linear time with respect to the number of states in the model. Our experiments demonstrate that the technique works in real-time, and is able to produce sound long-term predictions of people motion.
---
paper_title: Novel planning-based algorithms for human motion prediction

paper_content:

Human motion prediction from visual tracking is a challenging problem with a wide array of applications such as robotics, video surveillance and situation understanding. Recently, planning-based methods -which assume that people move by planning over a cost function- have emerged as one of the most promising alternatives. Nevertheless, state of the art planning based algorithms have shortcomings regarding their computational complexity and ability to predict for arbitrary time intervals. This paper addresses these shortcomings by leveraging alternative planning techniques (Fast Marching Method) and formulating efficient algorithms for goal estimation and full spatiotemporal prediction with lower complexity than comparable approaches. In preliminary experiments, the proposed method significantly outperforms the accuracy of the current state-of-the-art approach while reducing the computation time by a factor of 30 using a parallel version of our algorithm.
---
paper_title: Pedestrian path prediction using body language traits

paper_content:

Driver Assistance Systems have achieved a high level of maturity in the latest years. As an example of that, sophisticated pedestrian protection systems are already available in a number of commercial vehicles from several OEMs. However, accurate pedestrian path prediction is needed in order to go a step further in terms of safety and reliability, since it can make the difference between effective and non-effective intervention. In this paper, we consider the three-dimensional pedestrian body language in order to perform path prediction in a probabilistic framework. For this purpose, the different body parts and joints are detected using stereo vision. We propose the use of GPDM (Gaussian Process Dynamical Models) for reducing the high dimensionality of the input feature vector (composed by joints and displacement vectors) in the 3D pose space and for learning the pedestrian dynamics in a latent space. Experimental results show that accurate path prediction can be achieved at a time horizon of ≈ 0.8 s.
---
paper_title: Goal-Directed Pedestrian Prediction

paper_content:

Recent advances in road safety have lead to a constant decline of injured traffic participants in Europe per year. Still, the number of injured pedestrians remains nearly constant. As a countermeasure, active pedestrian safety is the focus of current research, for which accurate pedestrian prediction is a prerequisite. In this scope, we propose a method for dynamics-and environment-based pedestrian prediction. We introduce the pedestrian's destination as a latent variable and thus convert the prediction problem into a planning problem. The planning is executed based on the current dynamics of the pedestrian. The distribution over the destinations is modeled using a Particle Filter. Experimental results show a significant improvement over existing approaches such as Kalman Filters.
---
paper_title: Predicting actions to act predictably: Cooperative partial motion planning with maximum entropy models

paper_content:

This paper reports on a data-driven motion planning approach for interaction-aware, socially-compliant robot navigation among human agents. Autonomous mobile robots navigating in workspaces shared with human agents require motion planning techniques providing seamless integration and smooth navigation in such. Smooth integration in mixed scenarios calls for two abilities of the robot: predicting actions of others and acting predictably for them. The former requirement requests trainable models of agent behaviors in order to accurately forecast their actions in the future, taking into account their reaction on the robot's decisions. A human-like navigation style of the robot facilitates other agents—most likely not aware of the underlying planning technique applied—to predict the robot motion vice versa, resulting in smoother joint navigation. The approach presented in this paper is based on a feature-based maximum entropy model and is able to guide a robot in an unstructured, real-world environment. The model is trained to predict joint behavior of heterogeneous groups of agents from onboard data of a mobile platform. We evaluate the benefit of interaction-aware motion planning in a realistic public setting with a total distance traveled of over 4 km. Interestingly the motion models learned from human-human interaction did not hold for robot-human interaction, due to the high attention and interest of pedestrians in testing basic braking functionality of the robot.
---
paper_title: Social LSTM: Human Trajectory Prediction in Crowded Spaces

paper_content:

Pedestrians follow different trajectories to avoid obstacles and accommodate fellow pedestrians. Any autonomous vehicle navigating such a scene should be able to foresee the future positions of pedestrians and accordingly adjust its path to avoid collisions. This problem of trajectory prediction can be viewed as a sequence generation task, where we are interested in predicting the future trajectory of people based on their past positions. Following the recent success of Recurrent Neural Network (RNN) models for sequence prediction tasks, we propose an LSTM model which can learn general human movement and predict their future trajectories. This is in contrast to traditional approaches which use hand-crafted functions such as Social forces. We demonstrate the performance of our method on several public datasets. Our model outperforms state-of-the-art methods on some of these datasets. We also analyze the trajectories predicted by our model to demonstrate the motion behaviour learned by our model.
---
paper_title: Driver and pedestrian awareness-based collision risk analysis

paper_content:

We present a novel approach for vehicle-pedestrian collision risk analysis that incorporates mutual situational awareness, a degree of potential motion coupling and the spatial layout of the environment. The approach uses a Dynamic Bayesian Network (DBN) for modeling the individual object paths; collision risk is subsequently computed by an intersection operation.
---
paper_title: Goal-directed pedestrian model for long-term motion prediction with application to robot motion planning

paper_content:

A probabilistic goal-directed model is proposed for pedestrian motion using navigation function and statistics of human motion gathered in the environment. In comparison with existing models, this model is both computationally inexpensive and does not fail when the optimal direction of motion in terms of this model is non-unique. We further introduce a Rapidly-Exploring Random Tree (RRT)-based path planner developed for planning in state-time space. With the help of an improved distance metric, the planner is much faster than RRT-Blossom [11] in complex maps. In an environment with 10 pedestrians, the planner and motion prediction combined can perform in near-real time.
---
paper_title: Socially-aware robot navigation: A learning approach

paper_content:

The ability to act in a socially-aware way is a key skill for robots that share a space with humans. In this paper we address the problem of socially-aware navigation among people that meets objective criteria such as travel time or path length as well as subjective criteria such as social comfort. Opposed to model-based approaches typically taken in related work, we pose the problem as an unsupervised learning problem. We learn a set of dynamic motion prototypes from observations of relative motion behavior of humans found in publicly available surveillance data sets. The learned motion prototypes are then used to compute dynamic cost maps for path planning using an any-angle A* algorithm. In the evaluation we demonstrate that the learned behaviors are better in reproducing human relative motion in both criteria than a Proxemics-based baseline method.
---
paper_title: Online trajectory prediction and planning for social robot navigation

paper_content:

This paper addresses the safe and legible navigation of mobile robots in multi-agent encounters. A novel motion model provides the basis to predict, plan and coordinate agent trajectories in intersection scenarios. The approach establishes an implicit, non-overt cooperation between the robot and humans by linking the prediction and planning of agent trajectories within a unified representation in terms of timed elastic bands. The planning process maintains multiple topological alternatives to resolve the encounter in a manner compliant with the implicit rules and objectives of human proxemics. The trajectory is obtained by optimizing the timed elastic band considering multiple conflicting objectives such as fastest path and minimal spatial separation among agents but also global proxemic aspects such as motion coherence within a group. Cooperation is achieved by coupling predicted and planned agent trajectories to eventually reach an implicit agreement of the agents on how to circumnavigate each other. The parameters of the cost functions of the underlying motion model are identified by inverse optimal control from a dataset of 73 recorded encounters with up to five humans and a total of 283 individual trajectories. Playback simulations of recorded encounters and experiments with a robot traversing a group of oncoming humans demonstrate the feasibility of the approach to resolve general proxemic encounters.
---
paper_title: A probabilistic model of human motion and navigation intent for mobile robot path planning

paper_content:

In order to effectively plan paths in environments inhabited by humans, robots must accurately predict human motion. Typical approaches to human prediction simply assume a constant velocity which is not always valid. This paper proposes to determine the likely navigation intent of humans and use that to predict human motion. Navigation intent is determined by the function and structure of the environment. Manually assigned functional places are combined with automatically extracted navigation way-points to define a number of likely navigation targets within the environment. To predict human motion toward these targets, a probabilistic model of human motion is proposed which is based on motion probability grids generated from observed motion. The models of human navigation intent and motion are integrated with an autonomous mobile robot system, with a laser range sensor detecting humans moving within the environment, and a path planning system. The models of human navigation intent and motion are verified using real captured human motion data from an office environment. Examples of human motion prediction are also presented.
---
paper_title: Gaussian process regression flow for analysis of motion trajectories

paper_content:

Recognition of motions and activities of objects in videos requires effective representations for analysis and matching of motion trajectories. In this paper, we introduce a new representation specifically aimed at matching motion trajectories. We model a trajectory as a continuous dense flow field from a sparse set of vector sequences using Gaussian Process Regression. Furthermore, we introduce a random sampling strategy for learning stable classes of motions from limited data. Our representation allows for incrementally predicting possible paths and detecting anomalous events from online trajectories. This representation also supports matching of complex motions with acceleration changes and pauses or stops within a trajectory. We use the proposed approach for classifying and predicting motion trajectories in traffic monitoring domains and test on several data sets. We show that our approach works well on various types of complete and incomplete trajectories from a variety of video data sets with different frame rates.
---
paper_title: Activity forecasting

paper_content:

We address the task of inferring the future actions of people from noisy visual input. We denote this task activity forecasting. To achieve accurate activity forecasting, our approach models the effect of the physical environment on the choice of human actions. This is accomplished by the use of state-of-the-art semantic scene understanding combined with ideas from optimal control theory. Our unified model also integrates several other key elements of activity analysis, namely, destination forecasting, sequence smoothing and transfer learning. As proof-of-concept, we focus on the domain of trajectory-based activity analysis from visual input. Experimental results demonstrate that our model accurately predicts distributions over future actions of individuals. We show how the same techniques can improve the results of tracking algorithms by leveraging information about likely goals and trajectories.
---
paper_title: MX-LSTM: Mixing Tracklets and Vislets to Jointly Forecast Trajectories and Head Poses

paper_content:

Recent approaches on trajectory forecasting use tracklets to predict the future positions of pedestrians exploiting Long Short Term Memory (LSTM) architectures. This paper shows that adding vislets, that is, short sequences of head pose estimations, allows to increase significantly the trajectory forecasting performance. We then propose to use vislets in a novel framework called MX-LSTM, capturing the interplay between tracklets and vislets thanks to a joint unconstrained optimization of full covariance matrices during the LSTM backpropagation. At the same time, MX-LSTM predicts the future head poses, increasing the standard capabilities of the long-term trajectory forecasting approaches. With standard head pose estimators and an attentional-based social pooling, MX-LSTM scores the new trajectory forecasting state-of-the-art in all the considered datasets (Zara01, Zara02, UCY, and TownCentre) with a dramatic margin when the pedestrians slow down, a case where most of the forecasting approaches struggle to provide an accurate solution.
---
paper_title: You'll never walk alone: Modeling social behavior for multi-target tracking

paper_content:

Object tracking typically relies on a dynamic model to predict the object's location from its past trajectory. In crowded scenarios a strong dynamic model is particularly important, because more accurate predictions allow for smaller search regions, which greatly simplifies data association. Traditional dynamic models predict the location for each target solely based on its own history, without taking into account the remaining scene objects. Collisions are resolved only when they happen. Such an approach ignores important aspects of human behavior: people are driven by their future destination, take into account their environment, anticipate collisions, and adjust their trajectories at an early stage in order to avoid them. In this work, we introduce a model of dynamic social behavior, inspired by models developed for crowd simulation. The model is trained with videos recorded from birds-eye view at busy locations, and applied as a motion model for multi-people tracking from a vehicle-mounted camera. Experiments on real sequences show that accounting for social interactions and scene knowledge improves tracking performance, especially during occlusions.
---
paper_title: Behavior estimation for a complete framework for human motion prediction in crowded environments

paper_content:

In the present work, we propose and validate a complete probabilistic framework for human motion prediction in urban or social environments. Additionally, we formulate a powerful and useful tool: the human motion behavior estimator. Three different basic behaviors have been detected: Aware, Balanced and Unaware. Our approach is based on the Social Force Model (SFM) and the intentionality prediction BHMIP. The main contribution of the present work is to make use of the behavior estimator for formulating a reliable prediction framework of human trajectories under the influence of dynamic crowds, robots, and in general any moving obstacle. Accordingly, we have demonstrated the great performance of our long-term prediction algorithm, in real scenarios, comparing to other prediction methods.
---
paper_title: Generating Long-term Trajectories Using Deep Hierarchical Networks

paper_content:

We study the problem of modeling spatiotemporal trajectories over long time horizons using expert demonstrations. For instance, in sports, agents often choose action sequences with long-term goals in mind, such as achieving a certain strategic position. Conventional policy learning approaches, such as those based on Markov decision processes, generally fail at learning cohesive long-term behavior in such high-dimensional state spaces, and are only effective when fairly myopic decision-making yields the desired behavior. The key difficulty is that conventional models are ``single-scale'' and only learn a single state-action policy. We instead propose a hierarchical policy class that automatically reasons about both long-term and short-term goals, which we instantiate as a hierarchical neural network. We showcase our approach in a case study on learning to imitate demonstrated basketball trajectories, and show that it generates significantly more realistic trajectories compared to non-hierarchical baselines as judged by professional sports analysts.
---
paper_title: Planning-based prediction for pedestrians

paper_content:

We present a novel approach for determining robot movements that efficiently accomplish the robot's tasks while not hindering the movements of people within the environment. Our approach models the goal-directed trajectories of pedestrians using maximum entropy inverse optimal control. The advantage of this modeling approach is the generality of its learned cost function to changes in the environment and to entirely different environments. We employ the predictions of this model of pedestrian trajectories in a novel incremental planner and quantitatively show the improvement in hindrance-sensitive robot trajectory planning provided by our approach.
---
paper_title: Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks

paper_content:

Understanding human motion behavior is critical for autonomous moving platforms (like self-driving cars and social robots) if they are to navigate human-centric environments. This is challenging because human motion is inherently multimodal: given a history of human motion paths, there are many socially plausible ways that people could move in the future. We tackle this problem by combining tools from sequence prediction and generative adversarial networks: a recurrent sequence-to-sequence model observes motion histories and predicts future behavior, using a novel pooling mechanism to aggregate information across people. We predict socially plausible futures by training adversarially against a recurrent discriminator, and encourage diverse predictions with a novel variety loss. Through experiments on several datasets we demonstrate that our approach outperforms prior work in terms of accuracy, variety, collision avoidance, and computational complexity.
---
paper_title: Multimodal Probabilistic Model-Based Planning for Human-Robot Interaction

paper_content:

This paper presents a method for constructing human-robot interaction policies in settings where multimodality, i.e., the possibility of multiple highly distinct futures, plays a critical role in decision making. We are motivated in this work by the example of traffic weaving, e.g., at highway on-ramps/off-ramps, where entering and exiting cars must swap lanes in a short distance-a challenging negotiation even for experienced drivers due to the inherent multimodal uncertainty of who will pass whom. Our approach is to learn multimodal probability distributions over future human actions from a dataset of human-human exemplars and perform real-time robot policy construction in the resulting environment model through massively parallel sampling of human responses to candidate robot action sequences. Direct learning of these distributions is made possible by recent advances in the theory of conditional variational autoencoders (CVAEs), whereby we learn action distributions simultaneously conditioned on the present interaction history, as well as candidate future robot actions in order to take into account response dynamics. We demonstrate the efficacy of this approach with a human-in-the-loop simulation of a traffic weaving scenario.
---
paper_title: Learning Motion Patterns of People for Compliant Robot Motion

paper_content:

Whenever people move through their environments they do not move randomly. Instead, they usually follow specific trajectories or motion patterns corresponding to their intentions. Knowledge about such patterns enables a mobile robot to robustly keep track of persons in its environment and to improve its behavior. In this paper we propose a technique for learning collections of trajectories that characterize typical motion patterns of persons. Data recorded with laser-range finders are clustered using the expectation maximization algorithm. Based on the result of the clustering process, we derive a hidden Markov model that is applied to estimate the current and future positions of persons based on sensory input. We also describe how to incorporate the probabilistic belief about the potential trajectories of persons into the path planning process of a mobile robot. We present several experiments carried out in different environments with a mobile robot equipped with a laser-range scanner and a camera system...
---
paper_title: Activity forecasting

paper_content:

We address the task of inferring the future actions of people from noisy visual input. We denote this task activity forecasting. To achieve accurate activity forecasting, our approach models the effect of the physical environment on the choice of human actions. This is accomplished by the use of state-of-the-art semantic scene understanding combined with ideas from optimal control theory. Our unified model also integrates several other key elements of activity analysis, namely, destination forecasting, sequence smoothing and transfer learning. As proof-of-concept, we focus on the domain of trajectory-based activity analysis from visual input. Experimental results demonstrate that our model accurately predicts distributions over future actions of individuals. We show how the same techniques can improve the results of tracking algorithms by leveraging information about likely goals and trajectories.
---
paper_title: Behavior estimation for a complete framework for human motion prediction in crowded environments

paper_content:

In the present work, we propose and validate a complete probabilistic framework for human motion prediction in urban or social environments. Additionally, we formulate a powerful and useful tool: the human motion behavior estimator. Three different basic behaviors have been detected: Aware, Balanced and Unaware. Our approach is based on the Social Force Model (SFM) and the intentionality prediction BHMIP. The main contribution of the present work is to make use of the behavior estimator for formulating a reliable prediction framework of human trajectories under the influence of dynamic crowds, robots, and in general any moving obstacle. Accordingly, we have demonstrated the great performance of our long-term prediction algorithm, in real scenarios, comparing to other prediction methods.
---
paper_title: Prediction of moving objects in dynamic environments using Kalman filters

paper_content:

In this work, we describe a framework for predicting future positions and orientation of moving obstacles in a time-varying environment using Kalman filtering techniques. No constraint are placed on the obstacles motion. The proposed algorithm can be used in a variety of applications, one of which is robot motion planning in time varying environments. The advantage of using this model compared to reported ones in the literature is it's ability to start the prediction process from the first time step without the need to wait for few time steps before starting the prediction process. Early experimental results are very encouraging.
---
paper_title: A Predictive Collision Avoidance Model for Pedestrian Simulation

paper_content:

We present a new local method for collision avoidance that is based on collision prediction. In our model, each pedestrian predicts possible future collisions with other pedestrians and then makes an efficient move to avoid them. Experiments show that the new approach leads to considerably shorter and less curved paths, ensuring smooth avoidance behaviour and visually compelling simulations. The method reproduces emergent behaviour like lane formation that have been observed in real crowds. The technique is easy to implement and is fast, allowing the simulation in real time of crowds of thousands of pedestrians.
---
paper_title: Trajectory prediction of cyclists using a physical model and an artificial neural network

paper_content:

This article presents two methods for predicting the trajectories of cyclists at an intersection and compares them to a Kalman Filter (KF) approach. The first method uses a physical model of cyclists to predict their future position. The second method is based on a polynomial least-squares approximation in combination with a multilayer perceptron artificial neural network and is able to predict the future position of cyclists independent of their motion type such as “Starting”, “Stopping”, “Waiting” or “Passing”. To evaluate the performance of the methods, 566 tracks (394 for training, 172 for testing) of uninstructed cyclists were recorded at a public intersection using a wide angle stereo camera system and laser scanners. Using the tracks as input data, the future trajectory was predicted for a time horizon of 2.5 s. For starting motions the prediction using the physical model leads to 27% more accurate positions than the KF approach for a forecast horizon of 2.5 s. The neural network shows a 34% more accurate result for starting and stopping motions and a similar result for waiting and passing motions.
---
paper_title: Nonlinear constrained tracking of targets on roads

paper_content:

Ground targets are constrained to move on the Earth's surface and are most likely to travel along a road network. For targets on road, their interaction with the environment and with each other particularly at intersections is more structured, thus useful to tracking algorithms. Indeed, the knowledge of terrain database and road maps can be used as constraints and incorporated into the tracking algorithms. In this paper, we set forth a nonlinear formulation of tracking multiple interacting targets along road networks and apply nonlinear filtering techniques (namely, the extended Kalman filter, the unscented Kalman filter, the particle filter, and a hybrid Kalman particle filter) to the problem for such scenarios as air-to-ground surveillance. Simulation results are present to illustrate the performance of the nonlinear filters with the nonlinear formulation as compared to conventional ones.
---
paper_title: People tracking with human motion predictions from social forces

paper_content:

For many tasks in populated environments, robots need to keep track of current and future motion states of people. Most approaches to people tracking make weak assumptions on human motion such as constant velocity or acceleration. But even over a short period, human behavior is more complex and influenced by factors such as the intended goal, other people, objects in the environment, and social rules. This motivates the use of more sophisticated motion models for people tracking especially since humans frequently undergo lengthy occlusion events. In this paper, we consider computational models developed in the cognitive and social science communities that describe individual and collective pedestrian dynamics for tasks such as crowd behavior analysis. In particular, we integrate a model based on a social force concept into a multi-hypothesis target tracker. We show how the refined motion predictions translate into more informed probability distributions over hypotheses and finally into a more robust tracking behavior and better occlusion handling. In experiments in indoor and outdoor environments with data from a laser range finder, the social force model leads to more accurate tracking with up to two times fewer data association errors.
---
paper_title: Social force model for pedestrian dynamics

paper_content:

It is suggested that the motion of pedestrians can be described as if they would be subject to ``social forces.'' These ``forces'' are not directly exerted by the pedestrians' personal environment, but they are a measure for the internal motivations of the individuals to perform certain actions (movements). The corresponding force concept is discussed in more detail and can also be applied to the description of other behaviors. In the presented model of pedestrian behavior several force terms are essential: first, a term describing the acceleration towards the desired velocity of motion; second, terms reflecting that a pedestrian keeps a certain distance from other pedestrians and borders; and third, a term modeling attractive effects. The resulting equations of motion of nonlinearly coupled Langevin equations. Computer simulations of crowds of interacting pedestrians show that the social force model is capable of describing the self-organization of several observed collective effects of pedestrian behavior very realistically.
---
paper_title: Kalman Filtering with State Constraints : A Survey of Linear and Nonlinear Algorithms

paper_content:

The Kalman filter is the minimum-variance state estimator for linear dynamic systems with Gaussian noise. Even if the noise is non-Gaussian, the Kalman filter is the best linear estimator. For nonlinear systems it is not possible, in general, to derive the optimal state estimator in closed form, but various modifications of the Kalman filter can be used to estimate the state. These modifications include the extended Kalman filter, the unscented Kalman filter, and the particle filter. Although the Kalman filter and its modifications are powerful tools for state estimation, we might have information about a system that the Kalman filter does not incorporate. For example, we may know that the states satisfy equality or inequality constraints. In this case we can modify the Kalman filter to exploit this additional information and get better filtering performance than the Kalman filter provides. This paper provides an overview of various ways to incorporate state constraints in the Kalman filter and its nonlinear modifications. If both the system and state constraints are linear, then all of these different approaches result in the same state estimate, which is the optimal constrained linear state estimate. If either the system or constraints are nonlinear, then constrained filtering is, in general, not optimal, and different approaches give different results.
---
paper_title: Motion prediction of moving objects based on autoregressive model

paper_content:

In this paper, we describe a framework for predicting future positions and orientation of moving obstacles in a time-varying environment using autoregressive model (ARM) with conditional maximum likelihood estimate of the model parameters. No constraints are placed on the obstacles motion. The proposed algorithm can be used in a variety of applications, one of which is robot motion planning in time varying environments.
---
paper_title: Set-Based Prediction of Pedestrians in Urban Environments Considering Formalized Traffic Rules

paper_content:

Set-based predictions can ensure the safety of planned motions, since they provide a bounded region which includes all possible future states of nondeterministic models of other traffic participants. However, while autonomous vehicles are tested in urban environments, a set-based prediction tailored to pedestrians does not exist yet. This paper addresses this problem and presents an approach for set-based predictions of pedestrians using reachability analysis. We obtain tight over-approximations of pedestrians' reachable occupancy by incorporating the dynamics of pedestrians, contextual information, and traffic rules. In addition, since pedestrians often disregard traffic rules, our constraints automatically adapt so that such behaviors are included in the prediction. Using datasets of recorded pedestrians, we validate our proposed method and demonstrate its use for evasive maneuver planning of automated vehicles.
---
paper_title: A new model and efficient tracker for a target with curvilinear motion

paper_content:

A new model for targets moving in two dimensions is presented. Target motion is modelled as constant velocity with occasional along-track and cross-track acceleration inputs. Simulations show that a tracker based on this new model produces estimates superior to both the variable-dimension filter (VDF) and the circular-turn-based interacting multiple-model (IMM) tracker, By using nested multiple models, a modified tracker is created which has considerable computational savings compared with a standard IMM tracker.
---
paper_title: Pedestrian Group Behavior in a Cellular Automaton

paper_content:

The study of crowd movement is very important for planning mass events and evacuations. Although many potentially critical incidents are hardly predictable, others like clogging, can be anticipated. Studies have shown that pedestrians in social groups frequently contribute the biggest part to crowds, and that these groups have a significant impact on crowd movement. Furthermore social cooperative behaviour does not stop in emergency situations, but continues or even gets stronger. We employ a cellular automaton with attractive and repulsive potentials for the simulation of individual pedestrians. Then we explain how to integrate a concise model for social groups suitable for the basic modelling techniques. Further aspects of group behavior are discussed, namely large groups and the behavior after a group separation, and we propose a model for these. These investigations lead us to the conclusion that there is a growing need of agent based modelling when facing advanced aspects of pedestrian crowd behavior.
---
paper_title: Where will the oncoming vehicle be the next second?

paper_content:

A new image based approach for fast and robust tracking of vehicles from a moving platform is presented. Position, orientation, and the full motion state including velocity, acceleration, and yaw rate of a detected vehicle are estimated from a tracked 3D point cloud. This point cloud is computed by analyzing image sequences in both space and time, i.e. by fusion of stereo vision and tracked optical flow vectors. Starting from an automated initial vehicle hypothesis, the tracking is performed by means of Extended Kalman Filter. The filter combines the knowledge of where a point in the rigid point cloud has moved within a given time interval, with the dynamic model of a vehicle. The proposed system is applied to predict the driving path of other traffic participants and runs currently at 25 Hz (VGA images) on our demonstrator vehicle UTA.
---
paper_title: Improving data association by joint modeling of pedestrian trajectories and groupings

paper_content:

We consider the problem of data association in a multiperson tracking context. In semi-crowded environments, people are still discernible as individually moving entities, that undergo many interactions with other people in their direct surrounding. Finding the correct association is therefore difficult, but higher-order social factors, such as group membership, are expected to ease the problem. However, estimating group membership is a chicken-and-egg problem: knowing pedestrian trajectories, it is rather easy to find out possible groupings in the data, but in crowded scenes, it is often difficult to estimate closely interacting trajectories without further knowledge about groups. To this end, we propose a third-order graphical model that is able to jointly estimate correct trajectories and group memberships over a short time window. A set of experiments on challenging data underline the importance of joint reasoning for data association in crowded scenarios.
---
paper_title: Comparison and evaluation of advanced motion models for vehicle tracking

paper_content:

The estimation of a vehiclepsilas dynamic state is one of the most fundamental data fusion tasks for intelligent traffic applications. For that, motion models are applied in order to increase the accuracy and robustness of the estimation. This paper surveys numerous (especially curvilinear) models and compares their performance using a tracking tasks which includes the fusion of GPS and odometry data with an Unscented Kalman Filter. For evaluation purposes, a highly accurate reference trajectory has been recorded using an RTK-supported DGPS receiver. With this ground truth data, the performance of the models is evaluated in different scenarios and driving situations.
---
paper_title: BRVO: Predicting pedestrian trajectories using velocity-space reasoning

paper_content:

We introduce a novel, online method to predict pedestrian trajectories using agent-based velocity-space reasoning for improved human-robot interaction and collision-free navigation. Our formulation uses velocity obstacles to model the trajectory of each moving pedestrian in a robot's environment and improves the motion model by adaptively learning relevant parameters based on sensor data. The resulting motion model for each agent is computed using statistical inferencing techniques, including a combination of ensemble Kalman filters and a maximum-likelihood estimation algorithm. This allows a robot to learn individual motion parameters for every agent in the scene at interactive rates. We highlight the performance of our motion prediction method in real-world crowded scenarios, compare its performance with prior techniques, and demonstrate the improved accuracy of the predicted trajectories. We also adapt our approach for collision-free robot navigation among pedestrians based on noisy data and highlight the results in our simulator.
---
paper_title: Trajectory analysis and prediction for improved pedestrian safety: Integrated framework and evaluations

paper_content:

This paper presents a monocular and purely vision based pedestrian trajectory tracking and prediction framework with integrated map-based hazard inference. In Advanced Driver Assistance systems research, a lot of effort has been put into pedestrian detection over the last decade, and several pedestrian detection systems are indeed showing impressive results. Considerably less effort has been put into processing the detections further. We present a tracking system for pedestrians, which based on detection bounding boxes tracks pedestrians and is able to predict their positions in the near future. The tracking system is combined with a module which, based on the car's GPS position acquires a map and uses the road information in the map to know where the car can drive. Then the system warns the driver about pedestrians at risk, by combining the information about hazardous areas for pedestrians with a probabilistic position prediction for all observed pedestrians.
---
paper_title: Learning intentions for improved human motion prediction

paper_content:

For many tasks, robots need to operate in human populated environments. Human motion prediction therefore is gaining importance. The concept of social forces defines virtual repelling and attracting forces from and to obstacles and points of interest. These social forces can then be used to model typical human movements given an environment and a person's intention. This work shows how such models can exploit typical motion patterns summarized by growing hidden Markov models (GHMMs) that can be learned from data online and without human intervention. An extensive series of experiments shows that exploiting the intended position estimated using a GHMM within a social forces based motion model yields a significant performance gain in comparison with the standard constant velocity-based models.
---
paper_title: Modelling subgroup behaviour in crowd dynamics DEM simulation

paper_content:

This paper investigates the behaviour of subgroups in crowd dynamics by means of filming and observation. An existing crowd modelling program, CrowdDMX, based on a discrete element model (DEM) has been modified on the basis of observations made in this paper and literature. Each person is represented as three overlapping circles and motion is modelled in a Newtonian manner. It incorporates psychological forces as well as physical forces in a 2D time-stepping environment. The DEM model was modified to include realistic subgroup behaviour, representing people in the crowd desiring to stay together (families, friends, etc.). Subgroup psychological forces were incorporated. The previous model only simulated individuals moving independently, which was unrealistic in some situations as shown by the observation and filming part of the study. The revised program models subgroups realistically including the tendency to avoid subgroup division in cases of contra-flow.
---
paper_title: Who are you with and where are you going?

paper_content:

We propose an agent-based behavioral model of pedestrians to improve tracking performance in realistic scenarios. In this model, we view pedestrians as decision-making agents who consider a plethora of personal, social, and environmental factors to decide where to go next. We formulate prediction of pedestrian behavior as an energy minimization on this model. Two of our main contributions are simple, yet effective estimates of pedestrian destination and social relationships (groups). Our final contribution is to incorporate these hidden properties into an energy formulation that results in accurate behavioral prediction. We evaluate both our estimates of destination and grouping, as well as our accuracy at prediction and tracking against state of the art behavioral model and show improvements, especially in the challenging observational situation of infrequent appearance observations–something that might occur in thousands of webcams available on the Internet.
---
paper_title: Velocity-Based Driver Intent Inference at Urban Intersections in the Presence of Preceding Vehicles

paper_content:

Predicting turn and stop maneuvers of potentially errant drivers is a basic requirement for advanced driver assistance systems for urban intersections. Previous work has shown that an early estimate of the driver's intent can be inferred by evaluating the vehicle's speed during the intersection approach. In the presence of a preceding vehicle, however, the velocity profile might be dictated by car-following behavior rather than by the need to slow down before doing a left or right turn. To infer the driver's intent under such circumstances, a simple, real-time capable approach using a parametric model to represent both car-following and turning behavior is proposed. The performance of two alternative parameterizations based on observations at an individual intersection and a generic curvature-based model is evaluated in combination with two different Bayes net classification algorithms. In addition, the driver model is shown to be capable of predicting the future trajectory of the vehicle.
---
paper_title: Long-term Path Prediction in Urban Scenarios using Circular Distributions

paper_content:

Abstract Human ability to foresee the near future plays a key role in everyone's life to prevent potentially dangerous situations. To be able to make predictions is crucial when people have to interact with the surrounding environment. Modeling such capability can lead to the design of automated warning systems and provide moving robots with an intelligent way of interaction with changing situation. In this work we focus on a typical urban human-scene where we aim at predicting an agent's behavior using a stochastic model. In this approach, we fuse the various factors that would contribute to a human motion in different contexts. Our method uses previously observed trajectories to build point-wise circular distributions that after combination, provide a statistical smooth prediction towards the most likely areas. More specifically, a ray-launching procedure, based on a semantic segmentation, gives a coarse scene representation for collision avoidance; a nearly-constant velocity dynamic model smooths the acceleration progression and knowledge of the agent's destination may further steer the path prediction. Experimental results in structured scenes, validate the effectiveness of the method in predicting paths in comparison to actual trajectories.
---
paper_title: Congested traffic states in empirical observations and microscopic simulations, Phys

paper_content:

We present data from several German freeways showing different kinds of congested traffic forming near road inhomogeneities, specifically lane closings, intersections, or uphill gradients. The states are localized or extended, homogeneous or oscillating. Combined states are observed as well, like the coexistence of moving localized clusters and clusters pinned at road inhomogeneities, or regions of oscillating congested traffic upstream of nearly homogeneous congested traffic. The experimental findings are consistent with a recently proposed theoretical phase diagram for traffic near on-ramps [D. Helbing, A. Hennecke, and M. Treiber, Phys. Rev. Lett. 82, 4360 (1999)]. We simulate these situations with a continuous microscopic single-lane model, the ``intelligent driver model,'' using empirical boundary conditions. All observations, including the coexistence of states, are qualitatively reproduced by describing inhomogeneities with local variations of one model parameter. We show that the results of the microscopic model can be understood by formulating the theoretical phase diagram for bottlenecks in a more general way. In particular, a local drop of the road capacity induced by parameter variations has essentially the same effect as an on-ramp.
---
paper_title: Multiple target tracking in world coordinate with single, minimally calibrated camera

paper_content:

Tracking multiple objects is important in many application domains. We propose a novel algorithm for multi-object tracking that is capable of working under very challenging conditions such as minimal hardware equipment, uncalibrated monocular camera, occlusions and severe background clutter. To address this problem we propose a new method that jointly estimates object tracks, estimates corresponding 2D/3D temporal trajectories in the camera reference system as well as estimates the model parameters (pose, focal length, etc) within a coherent probabilistic formulation. Since our goal is to estimate stable and robust tracks that can be univocally associated to the object IDs, we propose to include in our formulation an interaction (attraction and repulsion) model that is able to model multiple 2D/3D trajectories in space-time and handle situations where objects occlude each other. We use a MCMC particle filtering algorithm for parameter inference and propose a solution that enables accurate and efficient tracking and camera model estimation. Qualitative and quantitative experimental results obtained using our own dataset and the publicly available ETH dataset shows very promising tracking and camera estimation results.
---
paper_title: Pedestrian Reactive Navigation for Crowd Simulation: a Predictive Approach Abstract

paper_content:

This paper addresses the problem of virtual pedestrian autonomous navigation for crowd simulation. It describes a method for solving interactions between pedestrians and avoiding inter-collisions. Our approach is agent-based and predictive: each agent perceives surrounding agents and extrapolates their trajectory in order to react to potential collisions. We aim at obtaining realistic results, thus the proposed model is calibrated from experimental motion capture data. Our method is shown to be valid and solves major drawbacks compared to previous approaches such as oscillations due to a lack of anticipation. We first describe the mathematical representation used in our model, we then detail its implementation, and finally, its calibration and validation from real data.
---
paper_title: Map-based long term motion prediction for vehicles in traffic environments

paper_content:

Depending on driver intention and current motion state of vehicle, an infinite set of possible future trajectories exists. In this paper we present a stochastic filter which is able to select a representative set of reasonable trajectories from this solution set using additional information from a digital map. This is achieved by representing the map's traffic lanes by their corresponding centerlines. Each detected vehicle is projected on these centerlines, where the necessary information is modeled as a multivariate random variable. With this stochastic model of traffic lanes and the stochastic vector of vehicle detection we define a new stochastic residual vector, which is used both in the lane assignment and during the generation of the motion predictions. For each traffic lane, which is assigned as relevant for a detected vehicle, we generate a new motion hypothesis by using an Extended Kalman Filter. To assess the plausibility of each motion hypothesis we employ an adaptive multivariate MCUSUM algorithm. The implemented stochastic filter is tested by evaluating real data from the Ko-FAS Research Initiative. While traditional motion prediction based on previous and current motion states of a vehicle only provide plausible prediction for a short term time horizon, we show that our approach achieves a reasonable motion hypothesis for long prediction intervals especially in complex scenarios as road intersections.
---
paper_title: Fusion of Tracks with Road Constraints

paper_content:

This paper is concerned with tracking of ground targets on roads and investigates possible ways to improve target state estimation via fusing a target’s track with information about the road along which the target is traveling. A target track is estimated using a surveillance radar whereas a digital map provides the road network of the region under surveillance. When the information about roads is as accurate as (or even better than) radar measurements, it is desired naturally to incorporate such information (fusion) into target state estimation. In this paper, roads are modeled with analytic functions and their fusion with a target track is cast as linear or nonlinear state constraints in an optimization procedure. The constrained optimization is then solved with the Lagrangian multiplier, leading to a closed-form solution for linear constraints and an iterative solution for second-order nonlinear constraints. Geometric interpretations of the solutions are provided for special cases. Compared to other methods, the track-to-road fusion using the constrained optimization technique can be easily implemented as an add-on module without changes to an existing tracker. For curved roads with coarse waypoints, the nonlinear constrained solution outperforms the piecewise linearized constrained approach. Computer simulation results are presented to illustrate the algorithms.
---
paper_title: Simulating and Evaluating the Local Behavior of Small Pedestrian Groups

paper_content:

Recent advancements in local methods have significantly improved the collision avoidance behavior of virtual characters. However, existing methods fail to take into account that in real life pedestrians tend to walk in small groups, consisting mainly of pairs or triples of individuals. We present a novel approach to simulate the walking behavior of such small groups. Our model describes how group members interact with each other, with other groups and individuals. We highlight the potential of our method through a wide range of test-case scenarios. We evaluate the results from our simulations using a number of quantitative quality metrics, and also provide visual and numerical comparisons with video footages of real crowds.
---
paper_title: GLMP- realtime pedestrian path prediction using global and local movement patterns

paper_content:

We present a novel real-time algorithm to predict the path of pedestrians in cluttered environments. Our approach makes no assumption about pedestrian motion or crowd density, and is useful for short-term as well as long-term prediction. We interactively learn the characteristics of pedestrian motion and movement patterns from 2D trajectories using Bayesian inference. These include local movement patterns corresponding to the current and preferred velocities and global characteristics such as entry points and movement features. Our approach involves no precomputation and we demonstrate the real-time performance of our prediction algorithm on sparse and noisy trajectory data extracted from dense indoor and outdoor crowd videos. The combination of local and global movement patterns can improve the accuracy of long-term prediction by 12–18% over prior methods in high-density videos.
---
paper_title: The Walking Behaviour of Pedestrian Social Groups and Its Impact on Crowd Dynamics

paper_content:

Human crowd motion is mainly driven by self-organized processes based on local interactions among pedestrians. While most studies of crowd behaviour consider only interactions among isolated individuals, it turns out that up to 70% of people in a crowd are actually moving in groups, such as friends, couples, or families walking together. These groups constitute medium-scale aggregated structures and their impact on crowd dynamics is still largely unknown. In this work, we analyze the motion of approximately 1500 pedestrian groups under natural condition, and show that social interactions among group members generate typical group walking patterns that influence crowd dynamics. At low density, group members tend to walk side by side, forming a line perpendicular to the walking direction. As the density increases, however, the linear walking formation is bent forward, turning it into a V-like pattern. These spatial patterns can be well described by a model based on social communication between group members. We show that the V-like walking pattern facilitates social interactions within the group, but reduces the flow because of its "non-aerodynamic" shape. Therefore, when crowd density increases, the group organization results from a trade-off between walking faster and facilitating social exchange. These insights demonstrate that crowd dynamics is not only determined by physical constraints induced by other pedestrians and the environment, but also significantly by communicative, social interactions among individuals.
---
paper_title: Recognition of dangerous situations within a cooperative group of vehicles

paper_content:

We consider the recognition of dangerous situations in vehicle traffic. Unscented Kalman filters are used to predict vehicle trajectories within a short prediction horizon [t 0 , t 0 + Δt]. Based on this prediction, for each vehicle pair the mutual distance is computed for [t 0 , t 0 + Δt], whereby the distance accounts for the geometric distance, for the prediction uncertainties as well as for the spatial dimensions of the vehicles. If at least one of the mutual distances falls below a distance threshold ∈ within [t 0 , t 0 + Δt], then a dangerous situation arises for the cooperative group and may lead to an autonomous cooperative driving manoeuvre. This approach allows the usage of the system in a mixed environment (only some vehicles are cooperative and cognitive). Obstacles can also be handled. The key issues in this ongoing research work are the recognition and classification of dangerous situations and the formation of a cooperative group constituting an operational unit. A common relevant picture within a group coordinator fuses the necessary information from all cooperative vehicles of the group and forms the basis for situation recognition and classification. This paper is a step to expand a Cooperative Collision Warning System (CCWS) to an integrated Cooperative Collision Avoidance and Cooperative Collision Mitigation System (CCAMS).
---
paper_title: Human Motion Behaviour Aware Planner (HMBAP) for path planning in dynamic human environments

paper_content:

For a robot navigating in a human inhabited dynamic environment, the knowledge of how the robot's movement can influence the trajectory of people around it can be very valuable. In this work we present a Human Motion Behaviour Aware Planner (HMBAP) which incorporates a Human Motion Behaviour Model (HMBM) in its planning stage to take advantage of this. HMBM is an obstacle avoidance model for people based on social forces which gives the robot an understanding of how people would react to its planned path. This information is useful for the robot to avoid imminent collisions with people in constricted spaces. The resulting robot behaviour is similar to how a human would move (in terms of avoidance behaviour) in a similar situation. We believe that this is a desirable feature for a robot navigating in a human inhabited environment. Our method shows good human-like navigation behaviour in situations where past methods fail to find a solution.
---
paper_title: Modeling local behavior for predicting social interactions towards human tracking

paper_content:

Human interaction dynamics are known to play an important role in the development of robust pedestrian trackers that are needed for a variety of applications in video surveillance. Traditional approaches to pedestrian tracking assume that each pedestrian walks independently and the tracker predicts the location based on an underlying motion model, such as a constant velocity or autoregressive model. Recent approaches have begun to leverage interaction, especially by modeling the repulsion forces among pedestrians to improve motion predictions. However, human interaction is more complex and is influenced by multiple social effects. This motivates the use of a more complex human interaction model for pedestrian tracking. In this paper, we propose a novel human tracking method by modeling complex social interactions. We present an algorithm that decomposes social interactions into multiple potential interaction modes. We integrate these multiple social interaction modes into an interactive Markov Chain Monte Carlo tracker and demonstrate how the developed method translates into a more informed motion prediction, resulting in robust tracking performance. We test our method on videos from unconstrained outdoor environments and evaluate it against common multi-object trackers.
---
paper_title: Learning Social Etiquette: Human Trajectory Understanding In Crowded Scenes

paper_content:

Humans navigate crowded spaces such as a university campus by following common sense rules based on social etiquette. In this paper, we argue that in order to enable the design of new target tracking or trajectory forecasting methods that can take full advantage of these rules, we need to have access to better data in the first place. To that end, we contribute a new large-scale dataset that collects videos of various types of targets (not just pedestrians, but also bikers, skateboarders, cars, buses, golf carts) that navigate in a real world outdoor environment such as a university campus. Moreover, we introduce a new characterization that describes the “social sensitivity” at which two targets interact. We use this characterization to define “navigation styles” and improve both forecasting models and state-of-the-art multi-target tracking–whereby the learnt forecasting models help the data association step.
---
paper_title: Walking Ahead: The Headed Social Force Model

paper_content:

Human motion models are finding an increasing number of novel applications in many different fields, such as building design, computer graphics and robot motion planning. The Social Force Model is one of the most popular alternatives to describe the motion of pedestrians. By resorting to a physical analogy, individuals are assimilated to point-wise particles subject to social forces which drive their dynamics. Such a model implicitly assumes that humans move isotropically. On the contrary, empirical evidence shows that people do have a preferred direction of motion, walking forward most of the time. Lateral motions are observed only in specific circumstances, such as when navigating in overcrowded environments or avoiding unexpected obstacles. In this paper, the Headed Social Force Model is introduced in order to improve the realism of the trajectories generated by the classical Social Force Model. The key feature of the proposed approach is the inclusion of the pedestrians' heading into the dynamic model used to describe the motion of each individual. The force and torque representing the model inputs are computed as suitable functions of the force terms resulting from the traditional Social Force Model. Moreover, a new force contribution is introduced in order to model the behavior of people walking together as a single group. The proposed model features high versatility, being able to reproduce both the unicycle-like trajectories typical of people moving in open spaces and the point-wise motion patterns occurring in high density scenarios. Extensive numerical simulations show an increased regularity of the resulting trajectories and confirm a general improvement of the model realism.
---
paper_title: Intentional motion on-line learning and prediction

paper_content:

Predicting motion of humans, animals and other objects which move according to internal plans is a challenging problem. Most existing approaches operate in two stages: (a) learning typical motion patterns by observing an environment and (b) predicting future motion on the basis of the learned patterns. In existing techniques, learning is performed off-line, hence, it is impossible to refine the existing knowledge on the basis of the new observations obtained during the prediction phase. We propose an approach which uses hidden Markov models (HMMs) to represent motion patterns. It is different from similar approaches because it is able to learn and predict in a concurrent fashion thanks to a novel approximate learning approach, based on the growing neural gas algorithm, which estimates both HMM parameters and structure. The found structure has the property of being a planar graph, thus enabling exact inference in linear time with respect to the number of states in the model. Our experiments demonstrate that the technique works in real-time, and is able to produce sound long-term predictions of people motion.
---
paper_title: Real-Time Motion Planning With Applications to Autonomous Urban Driving

paper_content:

This paper describes a real-time motion planning algorithm, based on the rapidly-exploring random tree (RRT) approach, applicable to autonomous vehicles operating in an urban environment. Extensions to the standard RRT are predominantly motivated by: 1) the need to generate dynamically feasible plans in real-time; 2) safety requirements; 3) the constraints dictated by the uncertain operating (urban) environment. The primary novelty is in the use of closed-loop prediction in the framework of RRT. The proposed algorithm was at the core of the planning and control software for Team MIT's entry for the 2007 DARPA Urban Challenge, where the vehicle demonstrated the ability to complete a 60 mile simulated military supply mission, while safely interacting with other autonomous and human driven vehicles.
---
paper_title: Goal-Directed Pedestrian Prediction

paper_content:

Recent advances in road safety have lead to a constant decline of injured traffic participants in Europe per year. Still, the number of injured pedestrians remains nearly constant. As a countermeasure, active pedestrian safety is the focus of current research, for which accurate pedestrian prediction is a prerequisite. In this scope, we propose a method for dynamics-and environment-based pedestrian prediction. We introduce the pedestrian's destination as a latent variable and thus convert the prediction problem into a planning problem. The planning is executed based on the current dynamics of the pedestrian. The distribution over the destinations is modeled using a Particle Filter. Experimental results show a significant improvement over existing approaches such as Kalman Filters.
---
paper_title: Threat assessment design for driver assistance system at intersections

paper_content:

This paper considers the decision-making problem for a human-driven vehicle crossing a road intersection in the presence of other, potentially errant, drivers. Our approach relies on a novel threat assessment module, which combines an intention predictor based on support vector machines with an efficient threat assessor using rapidly-exploring random trees. This module warns the host driver with the computed threat level and the corresponding best “escape maneuver” through the intersection, if the threat is sufficiently large. Through experimental results with small autonomous and human-driven vehicles, we demonstrate that this threat assessment module can be used in real-time to minimize the risk of collision.
---
paper_title: Experiment-based modeling, simulation and validation of interactions between virtual walkers

paper_content:

An interaction occurs between two humans when they walk with converging trajectories. They need to adapt their motion in order to avoid and cross one another at respectful distance. This paper presents a model for solving interactions between virtual humans. The proposed model is elaborated from experimental interactions data. We first focus our study on the pair-interaction case. In a second stage, we extend our approach to the multiple interactions case. Our experimental data allow us to state the conditions for interactions to occur between walkers, as well as each one's role during interaction and the strategies walkers set to adapt their motion. The low number of parameters of the proposed model enables its automatic calibration from available experimental data. We validate our approach by comparing simulated trajectories with real ones. We also provide comparison with previous solutions. We finally discuss the ability of our model to be extended to complex situations.
---
paper_title: Probabilistic long-term prediction for autonomous vehicles

paper_content:

Long-term prediction of traffic participants is crucial to enable autonomous driving on public roads. The quality of the prediction directly affects the frequency of trajectory planning. With a poor estimation of the future development, more computational effort has to be put in re-planning, and a safe vehicle state at the end of the planning horizon is not guaranteed. A holistic probabilistic prediction, considering inputs, results and parameters as random variables, highly reduces the problem. A time frame of several seconds requires a probabilistic description of the scene evolution, where uncertainty or accuracy is represented by the trajectory distribution. Following this strategy, a novel evaluation method is needed, coping with the fact, that the future evolution of a scene is also uncertain. We present a method to evaluate the probabilistic prediction of real traffic scenes with varying start conditions. The proposed prediction is based on a particle filter, estimating behavior describing parameters of a microscopic traffic model. Experiments on real traffic data with random leading vehicles show the applicability in terms of convergence, enabling long-term prediction using forward propagation.
---
paper_title: You'll never walk alone: Modeling social behavior for multi-target tracking

paper_content:

Object tracking typically relies on a dynamic model to predict the object's location from its past trajectory. In crowded scenarios a strong dynamic model is particularly important, because more accurate predictions allow for smaller search regions, which greatly simplifies data association. Traditional dynamic models predict the location for each target solely based on its own history, without taking into account the remaining scene objects. Collisions are resolved only when they happen. Such an approach ignores important aspects of human behavior: people are driven by their future destination, take into account their environment, anticipate collisions, and adjust their trajectories at an early stage in order to avoid them. In this work, we introduce a model of dynamic social behavior, inspired by models developed for crowd simulation. The model is trained with videos recorded from birds-eye view at busy locations, and applied as a motion model for multi-people tracking from a vehicle-mounted camera. Experiments on real sequences show that accounting for social interactions and scene knowledge improves tracking performance, especially during occlusions.
---
paper_title: A probabilistic model for estimating driver behaviors and vehicle trajectories in traffic environments

paper_content:

This paper presents a filter that is able to simultaneously estimate the behaviors of traffic participants and anticipate their future trajectories. This is achieved by recognizing the type of situation derived from the local situational context, which subsumes all information relevant for the drivers decision making. By explicitly taking into account the interactions between vehicles, it achieves a comprehensive situational understanding, inevitable for autonomous vehicles and driver assistance systems. This provides the necessary information for safe behavior decision making or motion planning. The filter is modeled as a Dynamic Bayesian Network. The factored state space, modeling the causal dependencies, allows to describe the models in a compact fashion and reduces the computational complexity of the inference process. The filter is evaluated in the context of a highway scenario, showing a good performance even with very noisy measurements. The presented framework is intended to be used in traffic environments but can be easily transferred to other robotic domains.
---
paper_title: VS-IMM using road map information for a ground target tracking

paper_content:

Tracking a ground target, based on GMTI radar measurements, is a challenging problem. This is due particularly to the environment with a high density road traffic, terrain masks... etc, and to the high manoeuverability of ground targets. In this paper, the tracking of a manoeuvring ground target on a road in a cluttered environment is studied using map information to enhance the tracking quality. Based on road segments positions, dynamic models under road constraints are built and an optimized projection technique of the estimated state is proposed in order to keep the target position and heading on the road. Using this projection approach, a multiple model tracking algorithm is derived and its performances are studied. The derived filter is an IMM (interacting multiple model) filter with a variable structure (VS-IMM) where the models set adapts sequentially according to the target position and to the road network configuration.
---
paper_title: Hybrid estimation of complex systems

paper_content:

Modern automated systems evolve both continuously and discretely, and hence require estimation techniques that go well beyond the capability of a typical Kalman filter. Multiple model (MM) estimation schemes track these system evolutions by applying a bank of filters, one for each discrete system mode. Modern systems, however, are often composed of many interconnected components that exhibit rich behaviors, due to complex, system-wide interactions. Modeling these systems leads to complex stochastic hybrid models that capture the large number of operational and failure modes. This large number of modes makes a typical MM estimation approach infeasible for online estimation. This paper analyzes the shortcomings of MM estimation, and then introduces an alternative hybrid estimation scheme that can efficiently estimate complex systems with large number of modes. It utilizes search techniques from the toolkit of model-based reasoning in order to focus the estimation on the set of most likely modes, without missing symptoms that might be hidden amongst the system noise. In addition, we present a novel approach to hybrid estimation in the presence of unknown behavioral modes. This leads to an overall hybrid estimation scheme for complex systems that robustly copes with unforeseen situations in a degraded, but fail-safe manner.
---
paper_title: Probabilistic Graphical Models: Principles and Techniques

paper_content:

Most tasks require a person or an automated system to reason -- to reach conclusions based on available information. The framework of probabilistic graphical models, presented in this book, provides a general approach for this task. The approach is model-based, allowing interpretable models to be constructed and then manipulated by reasoning algorithms. These models can also be learned automatically from data, allowing the approach to be used in cases where manually constructing a model is difficult or even impossible. Because uncertainty is an inescapable aspect of most real-world applications, the book focuses on probabilistic models, which make the uncertainty explicit and provide models that are more faithful to reality. Probabilistic Graphical Models discusses a variety of models, spanning Bayesian networks, undirected Markov networks, discrete and continuous models, and extensions to deal with dynamical systems and relational data. For each class of models, the text describes the three fundamental cornerstones: representation, inference, and learning, presenting both basic concepts and advanced techniques. Finally, the book considers the use of the proposed framework for causal reasoning and decision making under uncertainty. The main text in each chapter provides the detailed technical development of the key ideas. Most chapters also include boxes with additional material: skill boxes, which describe techniques; case study boxes, which discuss empirical cases related to the approach described in the text, including applications in computer vision, robotics, natural language understanding, and computational biology; and concept boxes, which present significant concepts drawn from the material in the chapter. Instructors (and readers) can group chapters in various combinations, from core topics to more technically advanced material, to suit their particular needs.
---
paper_title: Understanding interactions between traffic participants based on learned behaviors

paper_content:

Predicting vehicles' behaviors in a traffic scene can be very challenging due to many influences. Especially interactions with other traffic participants like vehicles or pedestrians are very crucial for the future movement while they are hard to model even with expert knowledge.
---
paper_title: Motion planning based on learning models of pedestrian and driver behaviors

paper_content:

Autonomous driving has shown the capability of providing driver convenience and enhancing safety. While introducing autonomous driving into our current traffic system, one significant issue is to make the autonomous vehicle be able to react in the same way as real human drivers. In order to ensure that an autonomous vehicle of the future will perform like human drivers, this paper proposes a vehicle motion planning model, which can represent how drivers control vehicles based on the assessment of traffic environments in the real signalized intersection. The proposed motion planning model comprises functions of pedestrian intention detection, gap detection and vehicle dynamic control. The three functions are constructed based on the analysis of actual data collected from real traffic environments. Finally, this paper demonstrates the performance of the proposed method by comparing the behaviors of our model with the behaviors of real pedestrians and human drivers. The experimental results show that our proposed model can achieve 85% recognition rate for the pedestrian crossing intention. Moreover, the vehicle controlled by the proposed motion planning model and the actual human-driven vehicle are highly similar with respect to the gap acceptance in intersections.
---
paper_title: Context-Based Path Prediction for Targets with Switching Dynamics

paper_content:

Anticipating future situations from streaming sensor data is a key perception challenge for mobile robotics and automated vehicles. We address the problem of predicting the path of objects with multiple dynamic modes. The dynamics of such targets can be described by a Switching Linear Dynamical System (SLDS). However, predictions from this probabilistic model cannot anticipate when a change in dynamic mode will occur. We propose to extract various types of cues with computer vision to provide context on the target’s behavior, and incorporate these in a Dynamic Bayesian Network (DBN). The DBN extends the SLDS by conditioning the mode transition probabilities on additional context states. We describe efficient online inference in this DBN for probabilistic path prediction, accounting for uncertainty in both measurements and target behavior. Our approach is illustrated on two scenarios in the Intelligent Vehicles domain concerning pedestrians and cyclists, so-called Vulnerable Road Users (VRUs). Here, context cues include the static environment of the VRU, its dynamic environment, and its observed actions. Experiments using stereo vision data from a moving vehicle demonstrate that the proposed approach results in more accurate path prediction than SLDS at the relevant short time horizon (1 s). It slightly outperforms a computationally more demanding state-of-the-art method.
---
paper_title: A Controlled Interactive Multiple Model Filter for Combined Pedestrian Intention Recognition and Path Prediction

paper_content:

We present a novel approach combining pedestrian intention recognition and path prediction for advanced video-based driver assistance systems. The core algorithm uses an Interacting Multiple Model Filter in combination with a Latent-dynamic Conditional Random Field model. The model integrates pedestrian dynamics and situational awareness using observations from a stereo-video system for pedestrian detection and human head pose estimation. Evaluation of our method is performed on a public available dataset addressing scenarios of lateral approaching pedestrians that might cross the road, turn into the road or stop at the curbside. During experiments, we demonstrate that the proposed approach leads to better path prediction performance in terms of a smaller lateral position error compared to state-of-the-art pedestrian intention recognition and path prediction approaches. The computational costs of our approach is comparatively low and therefore can be ported easily onto a real-time system.
---
paper_title: Pedestrian Path Prediction with Recursive Bayesian Filters: A Comparative Study

paper_content:

In the context of intelligent vehicles, we perform a comparative study on recursive Bayesian filters for pedestrian path prediction at short time horizons (< 2s). We consider Extended Kalman Filters (EKF) based on single dynamical models and Interacting Multiple Models (IMM) combining several such basic models (constant velocity/acceleration/turn). These are applied to four typical pedestrian motion types (crossing, stopping, bending in, starting). Position measurements are provided by an external state-of-the-art stereo vision-based pedestrian detector. We investigate the accuracy of position estimation and path prediction, and the benefit of the IMMs vs. the simpler single dynamical models. Special care is given to the proper sensor modeling and parameter optimization. The dataset and evaluation framework are made public to facilitate benchmarking.
---
paper_title: Towards a unified traffic situation estimation model — Street-dependent behaviour and motion models

paper_content:

For Advanced Driver Assistance Systems and Autonomous Driving, estimating and predicting traffic situations becomes more and more essential. Many approaches focus on one specific application like vehicle state estimation from sensor data or road model estimation from environment perception. To integrate single approaches to one coherent system, one unified model is needed where the existing applied algorithms can be grounded to. In this paper we propose a Unified Traffic Situation Estimation Model that describes the probabilistic dependencies between road elements. While its independence from time makes it usable for offline mapping tasks, we show that online prediction capabilities can be achieved by applying the model to a longitudinal vehicle state estimation problem: Using the Markov assumption and appropriate state spaces the general unified model can be specialized to an Interacting Multiple Model Filter. Finally, experiments show an improvement in state estimation and prediction over standard models, which only consider vehicle dynamics. Additionally the unified model allows the prediction of street related routes of vehicles.
---
paper_title: Ground target tracking with variable structure IMM estimator

paper_content:

In this paper we present the design of a Variable Structure Interacting Multiple Model (VS-IMM) estimator for tracking groups of ground targets on constrained paths using Moving Target Indicator (MTI) reports obtained from an airborne sensor. The targets are moving along a highway, with varying obscuration due to changing terrain conditions. In addition, the roads can branch, merge or cross-the scenario represents target convoys along a realistic road network with junctions, changing terrains, etc. Some of the targets may also move in an open field. This constrained motion estimation problem is handled using an IMM estimator with varying mode sets depending on the topography, The number of models in the IMM estimator, their types and their parameters are modified adaptively, in real-time, based on the estimated position of the target and the corresponding road/visibility conditions. This topography-based variable structure mechanism eliminates the need for carrying all the possible models throughout the entire tracking period as in the standard IMM estimator, significantly improving performance and reducing computational load. Data association is handled using an assignment algorithm. The estimator is designed to handle a very large number of ground targets simultaneously. A simulated scenario consisting of over one hundred targets is used to illustrate the selection of design parameters and the operation of the tracker. Performance measures are presented to contrast the benefits of the VS-IMM estimator over the Kalman filter and the standard IMM estimator, The VS-IMM estimator is then combined with multidimensional assignment to gain "time-depth." The additional benefit of using higher dimensional assignment algorithms for data association is also evaluated.
---
paper_title: Using road topology to improve cyclist path prediction

paper_content:

We learn motion models for cyclist path prediction on real-world tracks obtained from a moving vehicle, and propose to exploit the local road topology to obtain better predictive distributions. The tracks are extracted from the Tsinghua-Daimler Cyclist Benchmark for cyclist detection, and corrected for vehicle egomotion. Tracks are then spatially aligned to local curves and crossings in the road. We study a standard approach for path prediction in the literature based on Kalman Filters, as well as a mixture of specialized filters related to specific road orientations at junctions. Our experiments demonstrate an improved prediction accuracy (up to 20% on sharp turns) of mixing specialized motion models for canonical directions, and prior knowledge on the road topology. The new track data complements the existing video, disparity and annotation data of the original benchmark, and will be made publicly available.
---
paper_title: Stochastic reachable sets of interacting traffic participants

paper_content:

Knowledge about the future development of a certain road traffic situation is indispensable for safe path planning of autonomous ground vehicles or action selection of intelligent driver assistance systems. Due to a significant uncertainty about the future behavior of traffic participants, the prediction of traffic situations should be computed in a probabilistic setting. Under consideration of the dynamics of traffic participants, their future position is computed probabilistically by Markov chains that are obtained with methods known from hybrid verification. The characteristic feature of the presented approach is that all possible behaviors of traffic participants are considered, allowing to identify any dangerous future situation. The novel contribution of this work is the explicit modeling of the interaction of traffic participants, which leads to a more accurate prediction of their positions. Results are demonstrated for different traffic situations.
---
paper_title: Modeling and Prediction of Human Behavior

paper_content:

We propose that many human behaviors can be accurately described as a set of dynamic modes (e.g., Kalman filters) sequenced together by a Markov chain. We then use these dynamic Markov models to recognize human behaviors from sensory data and to predict human behaviors over a few seconds time. To test the power of this modeling approach, we report an experiment in which we were able to achieve 95% accuracy at predicting automobile drivers' subsequent actions from their initial preparatory movements. Language: en
---
paper_title: Estimation of Multivehicle Dynamics by Considering Contextual Information

paper_content:

Human drivers are endowed with an inborn ability to put themselves in the position of other drivers and reason about their behavior and intended actions. State-of-the-art driving-assistance systems, on the other hand, are generally limited to physical models and ad hoc safety rules. In order to drive safely amongst humans, autonomous vehicles need to develop an understanding of the situation in the form of a high-level description of the state of traffic participants. This paper presents a probabilistic model to estimate the state of vehicles by considering interactions between drivers immersed in traffic. The model is defined within a probabilistic filtering framework; estimation and prediction are carried out with statistical inference techniques. Memory requirements increase linearly with the number of vehicles, and thus, it is possible to scale the model to complex scenarios involving many participants. The approach is validated using real-world data collected by a group of interacting ground vehicles.
---
paper_title: Road occupancy prediction of traffic participants

paper_content:

We predict the road occupancy of traffic participants for collision avoidance systems. The occupancy sets are computed for consecutive time intervals and contain all reachable positions of traffic participants in compliance with a proposed dynamic model. Those sets make it possible to check if planned emergency maneuvers are collision-free in all possible future scenarios. However, no algorithm exists for exactly computing the occupancy when the model forbids unrealistic behavior such as leaving road boundaries or largely exceeding speed limits. For this reason, we provide methods to tightly overapproximate occupancy sets to ensure safe emergency maneuvers. We demonstrate the applicability of the approach by numerical examples, which show that the occupancy computation is not only efficient, but also tight enough to trigger emergency maneuvers almost at the last possible point in time.
---
paper_title: Interacting multiple model methods in target tracking: a survey

paper_content:

The Interacting Multiple Model (IMM) estimator is a suboptimal hybrid filter that has been shown to be one of the most cost-effective hybrid state estimation schemes. The main feature of this algorithm is its ability to estimate the state of a dynamic system with several behavior modes which can "switch" from one to another. In particular, the IMM estimator can be a self-adjusting variable-bandwidth filter, which makes it natural for tracking maneuvering targets. The importance of this approach is that it is the best compromise available currently-between complexity and performance: its computational requirements are nearly linear in the size of the problem (number of models) while its performance is almost the same as that of an algorithm with quadratic complexity. The objective of this work is to survey and put in perspective the existing IMM methods for target tracking problems. Special attention is given to the assumptions underlying each algorithm and its applicability to various situations.
---
paper_title: Reachability Analysis and its Application to the Safety Assessment of Autonomous Cars

paper_content:

This thesis is about the safety verification of dynamical systems using reachability analysis. Novel solutions have been developed for classical reachability analysis, stochastic reachability analysis, and their application to the safety assessment of autonomous cars. ::: ::: Classical reachability analysis computes the set of states that can be reached by a system. If the reachable set does not intersect any set of unsafe states, the safety of the system is guaranteed. Algorithms for this problem have been developed for linear, nonlinear, and hybrid systems. ::: ::: Stochastic reachability analysis measures the probability of reaching an unsafe set. One pursued approach computes over-approximative solutions for linear systems; another one generates a Markov chain which approximately computes the stochastic reachable set of arbitrary dynamics.
---
paper_title: IMM object tracking for high dynamic driving maneuvers

paper_content:

Classical object tracking approaches use a Kalman-filter with a single dynamic model which is therefore optimised to a single driving maneuver. In contrast the interacting multiple model (IMM) filter allows for several parallel models which are combined to a weighted estimate. Choosing models for different driving modes, such as constant speed, acceleration and strong acceleration changes, the object state estimation can be optimised for highly dynamic driving maneuvers. The paper describes the analysis of Stop&Go situations and the systematic parametrisation of the IMM method based on these statistics. The evaluation of the IMM approach is presented based on real sensor measurements of laser scanners, a radar and a video image processing unit.
---
paper_title: Driver and pedestrian awareness-based collision risk analysis

paper_content:

We present a novel approach for vehicle-pedestrian collision risk analysis that incorporates mutual situational awareness, a degree of potential motion coupling and the spatial layout of the environment. The approach uses a Dynamic Bayesian Network (DBN) for modeling the individual object paths; collision risk is subsequently computed by an intersection operation.
---
paper_title: Survey of maneuvering target tracking. Part V. Multiple-model methods

paper_content:

This is the fifth part of a series of papers that provide a comprehensive survey of techniques for tracking maneuvering targets without addressing the so-called measurement-origin uncertainty. Part I and Part II deal with target motion models. Part III covers measurement models and associated techniques. Part IV is concerned with tracking techniques that are based on decisions regarding target maneuvers. This part surveys the multiple-model methods $the use of multiple models (and filters) simultaneously - which is the prevailing approach to maneuvering target tracking in recent years. The survey is presented in a structured way, centered around three generations of algorithms: autonomous, cooperating, and variable structure. It emphasizes the underpinning of each algorithm and covers various issues in algorithm design, application, and performance.
---
paper_title: Vehicle Trajectory Prediction by Integrating Physics- and Maneuver-Based Approaches Using Interactive Multiple Models

paper_content:

Vehicle trajectory prediction helps automated vehicles and advanced driver-assistance systems have a better understanding of traffic environment and perform tasks such as criticality assessment in advance. In this study, an integrated vehicle trajectory prediction method is proposed by combining physics- and maneuver-based approaches. These two methods were combined for the reason that the physics-based trajectory prediction method could ensure accuracy in the short term with the consideration of vehicle running dynamic parameters, and the maneuver-based prediction approach has a long-term insight into future trajectories with maneuver estimation. In this study, the interactive multiple model trajectory prediction (IMMTP) method is proposed by combining the two predicting models. The probability of each model in the interactive multiple models could recursively adjust according to the predicting variance of each model. In addition, prediction uncertainty is considered by employing unscented Kalman filters in the physics-based prediction model. To the maneuver-based method, random elements for uncertainty are introduced to the trajectory of each maneuver inferred by using the dynamic Bayesian network. The approach is applied and analyzed in the lane-changing scenario by using naturalistic driving data. Comparison results indicate that IMMTP could achieve a more accurate prediction trajectory with a long prediction horizon.
---
paper_title: Improved state estimation through use of roads in ground tracking

paper_content:

The ability to provide an accurate view of a region of interest is standard among existing tracking systems. The extension of this capability to include accurate projections of the targets to future times increases the complexity of the problem. However, this ability to predict future locations is an important problem due to the inherent time latency that is present from sensor to shooter. Because of this, a major requirement of the tracking system is that it must be able to use available information to accurately pr edict future target locations. The focus of this paper is to describe an improved state estimation technique that incorporates road information by using a Variable Structure IMM. Furthermore, this approach will identify and account for stopped or stationary moving targets. Finally, some preliminary performance results will be presented.© (2000) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.
---
paper_title: Variable structure interacting multiple-model filter (VS-IMM) for tracking targets with transportation network constraints

paper_content:

A Ground Moving Target Indicator (GMTI) is developed using a Variable Structure Interacting Multiple Model Filter (VS-IMM). Current trackers use road network database information to either condition GMTI measurements, thereby altering the detection report, or constrain tracks formed on measurements near roads to the road network, thus making a hard decision about the location of the target. The VS-IMM allows for soft decisions about which road a target is possibly located on. The VS- IMM filter developed adaptively adds and deletes road models based upon history of measurement data or extrapolated tracks. As measurements are associated with existing tracks, a search of possible road segment models is performed to either add or delete road segment models as required. As targets on roads approach junctions, additional potential road segment models are added, as the targets pass the intersection only the most likely model is retained. When a target beginst o move into a road segment that is obscured, a model is added that modified the filter estimate and likelihood according to a hidden target model. The VS-IMM Filter includes two models for characterizing temporal vehicle movement. A model with low process noise is used to model targets that are traveling straight and a model with high process noise is used to model highly maneuvering targets. The state estimates from the possible road models are combined with the state estimate of the on-road and off-road multiple temporal models to produce the composite state estimate for the VS-IMM Filter.
---
paper_title: Conditional transition maps: Learning motion patterns in dynamic environments

paper_content:

In this paper we introduce a method for learning motion patterns in dynamic environments. Representations of dynamic environments have recently received an increasing amount of attention in the research community. Understanding dynamic environments is seen as one of the key challenges in order to enable autonomous navigation in real-world scenarios. However, representing the temporal dimension is a challenge yet to be solved. In this paper we introduce a spatial representation, which encapsulates the statistical dynamic behavior observed in the environment. The proposed Conditional Transition Map (CTMap) is a grid-based representation that associates a probability distribution for an object exiting the cell, given its entry direction. The transition parameters are learned from a temporal signal of occupancy on cells by using a local-neighborhood cross-correlation method. In this paper, we introduce the CTMap, the learning approach and present a proof-of-concept method for estimating future paths of dynamic objects, called Conditional Probability Propagation Tree (CPPTree). The evaluation is done using a real-world dataset collected at a busy roundabout.
---
paper_title: Stochastic prediction of human motion and control of robots in the service of human

paper_content:

The authors propose a control model for human cooperative robots. According to human motion which is measured by a human recognition system, future human position is predicted. Robot trajectories are changed by acceleration/deceleration/stop so as to minimize danger, which is computed by the predicted data. In this paper, a model of a stochastic process is adopted for the prediction because it can prevent inappropriate robot motion which might be caused by prediction error. A prediction plane is divided into square cells. Each cell has probability data by which cell number, direction and speed change. They are determined by the characteristics of motion of places beforehand. Computation of the stochastic state transition chain predicts the probability that a human exists in a cell at future time. Application to a laboratory room demonstrated that this method is effective for safe motion of human cooperative robots. >
---
paper_title: Knowledge Transfer for Scene-specific Motion Prediction

paper_content:

When given a single frame of the video, humans can not only interpret the content of the scene, but also they are able to forecast the near future. This ability is mostly driven by their rich prior knowledge about the visual world, both in terms of (i) the dynamics of moving agents, as well as (ii) the semantic of the scene. In this work we exploit the interplay between these two key elements to predict scene-specific motion patterns. First, we extract patch descriptors encoding the probability of moving to the adjacent patches, and the probability of being in that particular patch or changing behavior. Then, we introduce a Dynamic Bayesian Network which exploits this scene specific knowledge for trajectory prediction. Experimental results demonstrate that our method is able to accurately predict trajectories and transfer predictions to a novel scene characterized by similar elements.
---
paper_title: Voronoi tracking: location estimation using sparse and noisy sensor data

paper_content:

Tracking the activity of people in indoor environments has gained considerable attention in the robotics community over the last years. Most of the existing approaches are based on sensors, which allow to accurately determining the locations of people but do not provide means to distinguish between different persons. In this paper we propose a novel approach to tracking moving objects and their identity using noisy, sparse information collected by id-sensors such as infrared and ultrasound badge systems. The key idea of our approach is to use particle filters to estimate the locations of people on the Voronoi graph of the environment. By restricting particles to a graph, we make use of the inherent structure of indoor environments. The approach has two key advantages. First, it is by far more efficient and robust than unconstrained particle filters. Second, the Voronoi graph provides a natural discretization of human motion, which allows us to apply unsupervised learning techniques to derive typical motion patterns of the people in the environment. Experiments using a robot to collect ground-truth data indicate the superior performance of Voronoi tracking. Furthermore, we demonstrate that EM-based learning of behavior patterns increases the tracking performance and provides valuable information for high-level behavior recognition.
---
paper_title: Enabling Flow Awareness for Mobile Robots in Partially Observable Environments

paper_content:

Understanding the environment is a key requirement for any autonomous robot operation. There is extensive research on mapping geometric structure and perceiving objects. However, the environment is ...
---
paper_title: Real-Time Predictive Modeling and Robust Avoidance of Pedestrians with Uncertain, Changing Intentions

paper_content:

To plan safe trajectories in urban environments, autonomous vehicles must be able to quickly assess the future intentions of dynamic agents. Pedestrians are particularly challenging to model, as their motion patterns are often uncertain and/or unknown a priori. This paper presents a novel changepoint detection and clustering algorithm that, when coupled with offline unsupervised learning of a Gaussian process mixture model (DPGP), enables quick detection of changes in intent and online learning of motion patterns not seen in prior training data. The resulting long-term movement predictions demonstrate improved accuracy relative to offline learning alone, in terms of both intent and trajectory prediction. By embedding these predictions within a chance-constrained motion planner, trajectories which are probabilistically safe to pedestrian motions can be identified in real-time. Hardware experiments demonstrate that this approach can accurately predict pedestrian motion patterns from onboard sensor/perception data and facilitate robust navigation within a dynamic environment.
---
paper_title: Real-Time Certified Probabilistic Pedestrian Forecasting

paper_content:

The success of autonomous systems will depend upon their ability to safely navigate human-centric environments. This motivates the need for a real-time, probabilistic forecasting algorithm for pedestrians, cyclists, and other agents since these predictions will form a necessary step in assessing the risk of any action. This letter presents a novel approach to probabilistic forecasting for pedestrians based on weighted sums of ordinary differential equations that are learned from historical trajectory information within a fixed scene. The resulting algorithm is embarrassingly parallel and is able to work at real-time speeds using a naive Python implementation. The quality of predicted locations of agents generated by the proposed algorithm is validated on a variety of examples and is considerably higher than existing state of the art approaches over long time horizons.
---
paper_title: Modelling pedestrian trajectory patterns with Gaussian processes

paper_content:

We propose a non-parametric model for pedestrian motion based on Gaussian Process regression, in which trajectory data are modelled by regressing relative motion against current position. We show how the underlying model can be learned in an unsupervised fashion, demonstrating this on two databases collected from static surveillance cameras. We furthermore exemplify the use of model for prediction, comparing the recently proposed GP-Bayesfilters with a Monte Carlo method. We illustrate the benefit of this approach for long term motion prediction where parametric models such as Kalman Filters would perform poorly.
---
paper_title: Modeling Spatial-Temporal Dynamics of Human Movements for Predicting Future Trajectories

paper_content:

This paper presents a novel approach to modeling the dynamics of human movements with a grid-based representation. For each grid cell, we formulate the local dynamics using a variant of the left-to-right HMM, and thus explicitly model the exiting direction from the current cell. The dependency of this process on the entry direction is captured by employing the InputOutput HMM (IOHMM). On a higher level, we introduce the place where the whole trajectory originated into the IOHMM framework forming a hierarchical input structure. Therefore, we manage to capture both local spatial-temporal correlations and the long-term dependency on faraway initiating events, thus enabling the developed model to incorporate more information and to generate more informative predictions of future trajectories. The experimental results in an office corridor environment verify the capabilities of our method.
---
paper_title: Context-Aware Pedestrian Motion Prediction In Urban Intersections

paper_content:

This paper presents a novel context-based approach for pedestrian motion prediction in crowded, urban intersections, with the additional flexibility of prediction in similar, but new, environments. Previously, Chen et. al. combined Markovian-based and clustering-based approaches to learn motion primitives in a grid-based world and subsequently predict pedestrian trajectories by modeling the transition between learned primitives as a Gaussian Process (GP). This work extends that prior approach by incorporating semantic features from the environment (relative distance to curbside and status of pedestrian traffic lights) in the GP formulation for more accurate predictions of pedestrian trajectories over the same timescale. We evaluate the new approach on real-world data collected using one of the vehicles in the MIT Mobility On Demand fleet. The results show 12.5% improvement in prediction accuracy and a 2.65 times reduction in Area Under the Curve (AUC), which is used as a metric to quantify the span of predicted set of trajectories, such that a lower AUC corresponds to a higher level of confidence in the future direction of pedestrian motion.
---
paper_title: Incremental Learning of Statistical Motion Patterns With Growing Hidden Markov Models

paper_content:

Modeling and predicting human and vehicle motion is an active research do- main. Due to the difﬁculty of modeling the various factors that determine motion (eg in- ternal state, perception, etc.) this is often tackled by applying machine learning techniques to build a statistical model, using as input a collection of trajectories gathered through a sensor (eg camera, laser scanner), and then using that model to predict further motion. Un- fortunately, most current techniques use off-line learning algorithms, meaning that they are not able to learn new motion patterns once the learning stage has ﬁnished. In this paper, we present an approach which is able to learn new motion patterns incrementally, and in parallel with prediction. Our work is based on a novel extension to Hidden Markov Models called Growing Hidden Markov models.
---
paper_title: Augmented dictionary learning for motion prediction

paper_content:

Developing accurate models and efficient representations of multivariate trajectories is important for understanding the behavior patterns of mobile agents. This work presents a dictionary learning algorithm for developing a part-based trajectory representation, which combines merits of the existing Markovian-based and clustering-based approaches. In particular, this work presents the augmented semi-nonnegative sparse coding (ASNSC) algorithm for solving a constrained dictionary learning problem, and shows that the proposed method would converge to a local optimum given a convexity condition. We consider a trajectory modeling application, in which the learned dictionary atoms correspond to local motion patterns. Classical semi-nonnegative sparse coding approaches would add dictionary atoms with opposite signs to reduce the representational error, which can lead to learning noisy dictionary atoms that correspond poorly to local motion patterns. ASNSC addresses this problem and learns a concise set of intuitive motion patterns. ASNSC shows significant improvement over existing trajectory modeling methods in both prediction accuracy and computational time, as revealed by extensive numerical analysis on real datasets.
---
paper_title: Building a human behavior map from local observations

paper_content:

This paper presents a novel method for classifying regions from human movements in service robots' working environments. The entire space is segmented subject to the class type according to the functionality or affordance of each place which accommodates a typical human behavior. This is achieved based on a grid map in two steps. First a probabilistic model is developed to capture human movements for each grid cell by using a non-ergodic HMM. Then the learned transition probabilities corresponding to these movements are used to cluster all cells by using the K-means algorithm. The knowledge of typical human movements for each location, represented by the prototypes from K-means and summarized in a ‘behavior-based map’, enables a robot to adjust the strategy of interacting with people according to where they are located, and thus greatly enhances its capability to assist people. The performance of the proposed classification method is demonstrated by experimental results from 8 hours of data that are collected in a kitchen environment.
---
paper_title: A Bayesian nonparametric approach to modeling motion patterns

paper_content:

The most difficult--and often most essential--aspect of many interception and tracking tasks is constructing motion models of the targets. Experts rarely can provide complete information about a target's expected motion pattern, and fitting parameters for complex motion patterns can require large amounts of training data. Specifying how to parameterize complex motion patterns is in itself a difficult task. ::: ::: In contrast, Bayesian nonparametric models of target motion are very flexible and generalize well with relatively little training data. We propose modeling target motion patterns as a mixture of Gaussian processes (GP) with a Dirichlet process (DP) prior over mixture weights. The GP provides an adaptive representation for each individual motion pattern, while the DP prior allows us to represent an unknown number of motion patterns. Both automatically adjust the complexity of the motion model based on the available data. Our approach outperforms several parametric models on a helicopter-based car-tracking task on data collected from the greater Boston area.
---
paper_title: Camera-based observation of obstacle motions to derive statistical data for mobile robot motion planning

paper_content:

Mobile robots for advanced applications have to act in environments which contain moving obstacles. Motions of obstacles (e.g. humans) usually are not precisely predictable, but neither they are completely random. Long-term observation of obstacle behavior may yield knowledge about prevailing motion patterns. This paper presents concepts and results of an experimental system. Using cameras mounted at the ceiling, the workspace is observed. The image data is processed and transformed into a compact statistical representation of motion patterns. Mobile robot motion planning benefits from such additional knowledge: trajectories are rated in terms of collision probability and expected time for reaching the goal; planning yields efficient paths which are adapted to obstacle behavior.
---
paper_title: Modelling and Predicting Rhythmic Flow Patterns in Dynamic Environments

paper_content:

We present a time-dependent probabilistic map able to model and predict flow patterns of people in indoor environments. The proposed representation models the likelihood of motion direction on a grid-based map by a set of harmonic functions, which efficiently capture long-term (minutes to weeks) variations of crowd movements over time. The evaluation, performed on data from two real environments, shows that the proposed model enables prediction of human movement patterns in the future. Potential applications include human-aware motion planning, improving the efficiency and safety of robot navigation.
---
paper_title: Building Prior Knowledge: A Markov Based Pedestrian Prediction Model Using Urban Environmental Data

paper_content:

Autonomous Vehicles navigating in urban areas have a need to understand and predict future pedestrian behavior for safer navigation. This high level of situational awareness requires observing pedestrian behavior and extrapolating their positions to know future positions. While some work has been done in this field using Hidden Markov Models (HMMs), one of the few observed drawbacks of the method is the need for informed priors for learning behavior. In this work, an extension to the Growing Hidden Markov Model (GHMM) method is proposed to solve some of these drawbacks. This is achieved by building on existing work using potential cost maps and the principle of Natural Vision. As a consequence, the proposed model is able to predict pedestrian positions more precisely over a longer horizon compared to the state of the art. The method is tested over "legal" and "illegal" behavior of pedestrians, having trained the model with sparse observations and partial trajectories. The method, with no training data, is compared against a trained state of the art model. It is observed that the proposed method is robust even in new, previously unseen areas.
---
paper_title: A probabilistic model of human motion and navigation intent for mobile robot path planning

paper_content:

In order to effectively plan paths in environments inhabited by humans, robots must accurately predict human motion. Typical approaches to human prediction simply assume a constant velocity which is not always valid. This paper proposes to determine the likely navigation intent of humans and use that to predict human motion. Navigation intent is determined by the function and structure of the environment. Manually assigned functional places are combined with automatically extracted navigation way-points to define a number of likely navigation targets within the environment. To predict human motion toward these targets, a probabilistic model of human motion is proposed which is based on motion probability grids generated from observed motion. The models of human navigation intent and motion are integrated with an autonomous mobile robot system, with a laser range sensor detecting humans moving within the environment, and a path planning system. The models of human navigation intent and motion are verified using real captured human motion data from an office environment. Examples of human motion prediction are also presented.
---
paper_title: Sequence-to-Sequence Prediction of Vehicle Trajectory via LSTM Encoder-Decoder Architecture

paper_content:

In this paper, we propose a deep learning based vehicle trajectory prediction technique which can generate the future trajectory sequence of surrounding vehicles in real time. We employ the encoder-decoder architecture which analyzes the pattern underlying in the past trajectory using the long short-term memory (LSTM) based encoder and generates the future trajectory sequence using the LSTM based decoder. This structure produces the $K$ most likely trajectory candidates over occupancy grid map by employing the beam search technique which keeps the $K$ locally best candidates from the decoder output. The experiments conducted on highway traffic scenarios show that the prediction accuracy of the proposed method is significantly higher than the conventional trajectory prediction techniques.
---
paper_title: Intent Prediction of Pedestrians via Motion Trajectories Using Stacked Recurrent Neural Networks

paper_content:

The problem of intent understanding between highly and fully automated vehicles and vulnerable road users (VRUs) such as pedestrians in urban traffic environment has got some momentum over the past few years. Previous work has been tackling the problem using two common approaches, namely dynamical motion modeling and motion planning. In this paper, a novel radical end-to-end data-driven approach is proposed for long-term intent prediction of VRUs in urban traffic environment based solely on their motion trajectories. In the proposed approach, we utilized the widely adopted architecture of recurrent neural networks with Long-Short Term Memory (LSTM) modules to form a deep stacked LSTM network. Three common approaches used in the literature were compared against our proposed approach over two different real-world datasets involving pedestrians collected from vehicle-based stereo cameras. The results over the testing datasets showed that the proposed approach achieved higher accuracies over most of the scenarios of the testing datasets with a small mean lateral position error of 0.48 m. Moreover, the proposed approach showed also a significant generalization capability over totally unobserved testing scenes during the training phase with only 0.58 m in mean lateral position error.
---
paper_title: Mobile Agent Trajectory Prediction using Bayesian Nonparametric Reachability Trees

paper_content:

This paper presents an efficient trajectory prediction algorithm that has been developed to improve the performance of future collision avoidance and detection systems. The main idea is to embed the inferred intention information of surrounding agents into their estimated reachability sets to obtain a probabilistic description of their future paths. More specifically, the proposed approach combines the recently developed RRT-Reach algorithm and mixtures of Gaussian Processes. RRT-Reach was introduced by the authors as an extension of the closed-loop rapidly-exploring random tree (CL-RRT) algorithm to compute reachable sets of moving objects in real-time. A mixture of Gaussian processes (GP) is a flexible nonparametric Bayesian model used to represent a distribution over trajectories and have been previously demonstrated by the authors in a UAV interception and tracking of ground vehicles planning scheme. The mixture is trained using typical maneuvers learned from statistical data, and RRT-Reach utilizes samples from the GP to grow probabilistically weighted feasible paths of the surrounding vehicles. The resulting approach, denoted as RR-GP, has RRTReach’s benefits of computing trajectories that are dynamically feasible by construction, therefore efficiently approximating the reachability set of surrounding vehicles following typical patterns. RRT-GP also features the GP mixture’s benefits of providing a probabilistic weighting on the feasible trajectories produced by RRTReach, allowing our system to systematically weight trajectories by their likelihood. A demonstrative example on a car-like vehicle illustrates the advantages of the RR-GP approach by comparing it to two other GP-based algorithms.
---
paper_title: Trajectory prediction of cyclists using a physical model and an artificial neural network

paper_content:

This article presents two methods for predicting the trajectories of cyclists at an intersection and compares them to a Kalman Filter (KF) approach. The first method uses a physical model of cyclists to predict their future position. The second method is based on a polynomial least-squares approximation in combination with a multilayer perceptron artificial neural network and is able to predict the future position of cyclists independent of their motion type such as “Starting”, “Stopping”, “Waiting” or “Passing”. To evaluate the performance of the methods, 566 tracks (394 for training, 172 for testing) of uninstructed cyclists were recorded at a public intersection using a wide angle stereo camera system and laser scanners. Using the tracks as input data, the future trajectory was predicted for a time horizon of 2.5 s. For starting motions the prediction using the physical model leads to 27% more accurate positions than the KF approach for a forecast horizon of 2.5 s. The neural network shows a 34% more accurate result for starting and stopping motions and a similar result for waiting and passing motions.
---
paper_title: Context-Aware Trajectory Prediction

paper_content:

Human motion and behaviour in crowded spaces is influenced by several factors, such as the dynamics of other moving agents in the scene, as well as the static elements that might be perceived as points of attraction or obstacles. In this work, we present a new model for human trajectory prediction which is able to take advantage of both human-human and human-space interactions. The future trajectory of humans, are generated by observing their past positions and interactions with the surroundings. To this end, we propose a"context-aware"recurrent neural network LSTM model, which can learn and predict human motion in crowded spaces such as a sidewalk, a museum or a shopping mall. We evaluate our model on a public pedestrian datasets, and we contribute a new challenging dataset that collects videos of humans that navigate in a (real) crowded space such as a big museum. Results show that our approach can predict human trajectories better when compared to previous state-of-the-art forecasting models.
---
paper_title: Unfreezing the robot: Navigation in dense, interacting crowds

paper_content:

In this paper, we study the safe navigation of a mobile robot through crowds of dynamic agents with uncertain trajectories. Existing algorithms suffer from the “freezing robot” problem: once the environment surpasses a certain level of complexity, the planner decides that all forward paths are unsafe, and the robot freezes in place (or performs unnecessary maneuvers) to avoid collisions. Since a feasible path typically exists, this behavior is suboptimal. Existing approaches have focused on reducing the predictive uncertainty for individual agents by employing more informed models or heuristically limiting the predictive covariance to prevent this overcautious behavior. In this work, we demonstrate that both the individual prediction and the predictive uncertainty have little to do with the frozen robot problem. Our key insight is that dynamic agents solve the frozen robot problem by engaging in “joint collision avoidance”: They cooperatively make room to create feasible trajectories. We develop IGP, a nonparametric statistical model based on dependent output Gaussian processes that can estimate crowd interaction from data. Our model naturally captures the non-Markov nature of agent trajectories, as well as their goal-driven navigation. We then show how planning in this model can be efficiently implemented using particle based inference. Lastly, we evaluate our model on a dataset of pedestrians entering and leaving a building, first comparing the model with actual pedestrians, and find that the algorithm either outperforms human pedestrians or performs very similarly to the pedestrians. We also present an experiment where a covariance reduction method results in highly overcautious behavior, while our model performs desirably.
---
paper_title: Behavioral Priors for Detection and Tracking of Pedestrians in Video Sequences

paper_content:

In this paper we address the problem of detection and tracking of pedestrians in complex scenarios. The inclusion of prior knowledge is more and more crucial in scene analysis to guarantee flexibility and robustness, necessary to have reliability in complex scenes. We aim to combine image processing methods with behavioral models of pedestrian dynamics, calibrated on real data. We introduce Discrete Choice Models (DCM) for pedestrian behavior and we discuss their integration in a detection and tracking context. The obtained results show how it is possible to combine both methodologies to improve the performances of such systems in complex sequences.
---
paper_title: Generating Long-term Trajectories Using Deep Hierarchical Networks

paper_content:

We study the problem of modeling spatiotemporal trajectories over long time horizons using expert demonstrations. For instance, in sports, agents often choose action sequences with long-term goals in mind, such as achieving a certain strategic position. Conventional policy learning approaches, such as those based on Markov decision processes, generally fail at learning cohesive long-term behavior in such high-dimensional state spaces, and are only effective when fairly myopic decision-making yields the desired behavior. The key difficulty is that conventional models are ``single-scale'' and only learn a single state-action policy. We instead propose a hierarchical policy class that automatically reasons about both long-term and short-term goals, which we instantiate as a hierarchical neural network. We showcase our approach in a case study on learning to imitate demonstrated basketball trajectories, and show that it generates significantly more realistic trajectories compared to non-hierarchical baselines as judged by professional sports analysts.
---
paper_title: Cyclist Trajectory Prediction Using Bidirectional Recurrent Neural Networks

paper_content:

Predicting a long-term horizon of vulnerable road users’ trajectories such as cyclists become an inevitable task for a reliable operation of highly and fully automated vehicles. In the literature, this problem is often tackled using linear dynamics-based approaches based on recursive Bayesian filters. These approaches are usually challenged when it comes to predicting long-term horizon of trajectories (more than 1 sec). Additionally, they also have difficulties in predicting non-linear motions such as maneuvers done by cyclists in traffic environments. In this work, we are proposing two novel models based on deep stacked recurrent neural networks for the task of cyclists trajectories prediction to overcome some of the aforementioned challenges. Our proposed predictive models have achieved robust prediction results when evaluated on a real-life cyclist trajectories dataset collected using vehicle-based sensors in the urban traffic environment. Furthermore, our proposed models have outperformed other traditional approaches with an improvement of more than 50% in mean error score averaged over all the predicted cyclists’ trajectories.
---
paper_title: Road Infrastructure Indicators for Trajectory Prediction

paper_content:

Safe and comfortable path planning in a dynamic urban environment is essential to an autonomous vehicle. This requires the future trajectories of all other road users in the environment of the vehicle. These trajectories are predicted through modeling the motion and behaviour of these road users. In this work we state that for efficient trajectory prediction only motion indicators are not sufficient. Therefore, we propose using a curvilinear coordinate system with curvature as road infrastructure indicators to improve motion modeling and trajectory prediction. With experiments, we show that the curvilinear coordinate system with curvature sufficiently incorporates the road structure. Furthermore, we show that a sequence-tosequence RNN model is suitable to incorporate road curvature indicators directly into the modeling and prediction.
---
paper_title: 3DOF Pedestrian Trajectory Prediction Learned from Long-Term Autonomous Mobile Robot Deployment Data

paper_content:

This paper presents a novel 3DOF pedestrian trajectory prediction approach for autonomous mobile service robots. While most previously reported methods are based on learning of 2D positions in monocular camera images, our approach uses range-finder sensors to learn and predict 3DOF pose trajectories (i.e. 2D position plus 1D rotation within the world coordinate system). Our approach, T-Pose-LSTM (Temporal 3DOF-Pose Long-Short-Term Memory), is trained using long-term data from real-world robot deployments and aims to learn context-dependent (environment- and time-specific) human activities. Our approach incorporates long-term temporal information (i.e. date and time) with short-term pose observations as input. A sequence-to-sequence LSTM encoder-decoder is trained, which encodes observations into LSTM and then decodes as predictions. For deployment, it can perform on-the-fly prediction in real-time. Instead of using manually annotated data, we rely on a robust human detection, tracking and SLAM system, providing us with examples in a global coordinate system. We validate the approach using more than 15K pedestrian trajectories recorded in a care home environment over a period of three months. The experiment shows that the proposed T-Pose-LSTM model advances the state-of-the-art 2D-based method for human trajectory prediction in long-term mobile robot deployments.
---
paper_title: Bi-Prediction: Pedestrian Trajectory Prediction Based on Bidirectional LSTM Classification

paper_content:

Pedestrian trajectory prediction is important in various applications such as driverless vehicles, social robots, intelligent tracking systems and space planning. Existing methods focus on analysing the influence of neighbours but ignore the effect of the intended destinations of pedestrians which also plays a key role in route planning. In this paper, we propose a novel two- stage trajectory prediction method to yield multiple prediction trajectories with different probabilities towards different destination regions in the scene. Our method, which we refer to as Bi-Prediction, uses a bidirectional LSTM architecture to automatically classify trajectories into a small number of route classes before trajectory prediction. We have evaluated our method against two baseline methods and three state-of-art methods on two benchmark datasets. Our experimental results show that the extra classification stage improves the accuracy of the predicted trajectories.
---
paper_title: Probabilistic vehicle trajectory prediction over occupancy grid map via recurrent neural network

paper_content:

In this paper, we propose an efficient vehicle trajectory prediction framework based on recurrent neural network. Basically, the characteristic of the vehicle's trajectory is different from that of regular moving objects since it is affected by various latent factors including road structure, traffic rules, and driver's intention. Previous state of the art approaches use sophisticated vehicle behavior model describing these factors and derive the complex trajectory prediction algorithm, which requires a system designer to conduct intensive model optimization for practical use. Our approach is data-driven and simple to use in that it learns complex behavior of the vehicles from the massive amount of trajectory data through deep neural network model. The proposed trajectory prediction method employs the recurrent neural network called long short-term memory (LSTM) to analyze the temporal behavior and predict the future coordinate of the surrounding vehicles. The proposed scheme feeds the sequence of vehicles' coordinates obtained from sensor measurements to the LSTM and produces the probabilistic information on the future location of the vehicles over occupancy grid map. The experiments conducted using the data collected from highway driving show that the proposed method can produce reasonably good estimate of future trajectory.
---
paper_title: Multimodal Interaction-aware Motion Prediction for Autonomous Street Crossing

paper_content:

For mobile robots navigating on sidewalks, it is essential to be able to safely cross street intersections. Most existing approaches rely on the recognition of the traffic light signal to make an informed crossing decision. Although these approaches have been crucial enablers for urban navigation, the capabilities of robots employing such approaches are still limited to navigating only on streets containing signalized intersections. In this paper, we address this challenge and propose a multimodal convolutional neural network framework to predict the safety of a street intersection for crossing. Our architecture consists of two subnetworks; an interaction-aware trajectory estimation stream IA-TCNN, that predicts the future states of all observed traffic participants in the scene, and a traffic light recognition stream AtteNet. Our IA-TCNN utilizes dilated causal convolutions to model the behavior of the observable dynamic agents in the scene without explicitly assigning priorities to the interactions among them. While AtteNet utilizes Squeeze-Excitation blocks to learn a content-aware mechanism for selecting the relevant features from the data, thereby improving the noise robustness. Learned representations from the traffic light recognition stream are fused with the estimated trajectories from the motion prediction stream to learn the crossing decision. Furthermore, we extend our previously introduced Freiburg Street Crossing dataset with sequences captured at different types of intersections, demonstrating complex interactions among the traffic participants. Extensive experimental evaluations on public benchmark datasets and our proposed dataset demonstrate that our network achieves state-of-the-art performance for each of the subtasks, as well as for the crossing safety prediction.
---
paper_title: Learning spatio-temporal patterns for predicting object behaviour

paper_content:

Abstract Rule-based systems employed to model complex object behaviours, do not necessarily provide a realistic portrayal of true behaviour. To capture the real characteristics in a specific environment, a better model may be learnt from observation. This paper presents a novel approach to learning long-term spatio-temporal patterns of objects in image sequences, using a neural network paradigm to predict future behaviour. The results demonstrate the application of our approach to the problem of predicting animal behaviour in response to a predator.
---
paper_title: Pedestrian's Trajectory Forecast in Public Traffic with Artificial Neural Networks

paper_content:

This paper focuses on forecasting of pedestrian's short-time trajectories up to 2.5 s for traffic safety applications. We present a self-learning approach based on artificial neural network movement models and compare it to traditional constant velocity Kalman Filter prediction and extrapolation of polynomials fitted using a least-squares error. Trajectories of uninstructed pedestrians in public traffic at a real urban intersection are acquired by a wide angle stereo camera setup in combination with a 3D head tracking framework. Results using this real-world data show that the artificial neural network significantly improves forecast quality compared to other approaches especially for critical traffic scenes including velocity changes such as starting and stopping. For those velocity changes a reduction of position estimation errors of about 21% compared to the Kalman Filter and to extrapolation of polynomials is obtained. By means of a concrete pedestrian-vehicle scenario we demonstrate the benefit of the proposed approach for an advanced driver assistant system in terms of reaction time.
---
paper_title: Modeling cooperative navigation in dense human crowds

paper_content:

For robots to be a part of our daily life, they need to be able to navigate among crowds not only safely but also in a socially compliant fashion. This is a challenging problem because humans tend to navigate by implicitly cooperating with one another to avoid collisions, while heading toward their respective destinations. Previous approaches have used handcrafted functions based on proximity to model human-human and human-robot interactions. However, these approaches can only model simple interactions and fail to generalize for complex crowded settings. In this paper, we develop an approach that models the joint distribution over future trajectories of all interacting agents in the crowd, through a local interaction model that we train using real human trajectory data. The interaction model infers the velocity of each agent based on the spatial orientation of other agents in his vicinity. During prediction, our approach infers the goal of the agent from its past trajectory and uses the learned model to predict its future trajectory. We demonstrate the performance of our method against a state-of-the-art approach on a public dataset and show that our model outperforms when predicting future trajectories for longer horizons.
---
paper_title: Will the Pedestrian Cross? A Study on Pedestrian Path Prediction

paper_content:

Future vehicle systems for active pedestrian safety will not only require a high recognition performance but also an accurate analysis of the developing traffic situation. In this paper, we present a study on pedestrian path prediction and action classification at short subsecond time intervals. We consider four representative approaches: two novel approaches (based on Gaussian process dynamical models and probabilistic hierarchical trajectory matching) that use augmented features derived from dense optical flow and two approaches as baseline that use positional information only (a Kalman filter and its extension to interacting multiple models). In experiments using stereo vision data obtained from a vehicle, we investigate the accuracy of path prediction and action classification at various time horizons, the effect of various errors (image localization, vehicle egomotion estimation), and the benefit of the proposed approaches. The scenario of interest is that of a crossing pedestrian, who might stop or continue walking at the road curbside. Results indicate similar performance of the four approaches on walking motion, with near-linear dynamics. During stopping, however, the two newly proposed approaches, with nonlinear and/or higher order models and augmented motion features, achieve a more accurate position prediction of 10-50 cm at a time horizon of 0-0.77 s around the stopping event.
---
paper_title: Pedestrian Trajectory Prediction in Extremely Crowded Scenarios

paper_content:

Pedestrian trajectory prediction under crowded circumstances is a challenging problem owing to human interaction and the complexity of the trajectory pattern. Various methods have been proposed for solving this problem, ranging from traditional Bayesian analysis to Social Force model and deep learning methods. However, most existing models heavily depend on specific scenarios because the trajectory model is constructed in absolute coordinates even though the motion trajectory as well as human interaction are in relative motion. In this study, a novel trajectory prediction model is proposed to capture the relative motion of pedestrians in extremely crowded scenarios. Trajectory sequences and human interaction are first represented with relative motion and then integrated to our model to predict pedestrians’ trajectories. The proposed model is based on Long Short Term Memory (LSTM) structure and consists of an encoder and a decoder which are trained by truncated back propagation. In addition, an anisotropic neighborhood setting is proposed instead of traditional neighborhood analysis. The proposed approach is validated using trajectory data acquired at an extremely crowded train station in Tokyo, Japan. The trajectory prediction experiments demonstrated that the proposed method outperforms existing methods and is stable for predictions of varying length even when the model is trained with a controlled short trajectory sequence.
---
paper_title: Structural-RNN: Deep Learning on Spatio-Temporal Graphs

paper_content:

Deep Recurrent Neural Network architectures, though remarkably capable at modeling sequences, lack an intuitive high-level spatio-temporal structure. That is while many problems in computer vision inherently have an underlying high-level structure and can benefit from it. Spatio-temporal graphs are a popular tool for imposing such high-level intuitions in the formulation of real world problems. In this paper, we propose an approach for combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural Networks~(RNNs). We develop a scalable method for casting an arbitrary spatio-temporal graph as a rich RNN mixture that is feedforward, fully differentiable, and jointly trainable. The proposed method is generic and principled as it can be used for transforming any spatio-temporal graph through employing a certain set of well defined steps. The evaluations of the proposed approach on a diverse set of problems, ranging from modeling human motion to object interactions, shows improvement over the state-of-the-art with a large margin. We expect this method to empower new approaches to problem formulation through high-level spatio-temporal graphs and Recurrent Neural Networks.
---
paper_title: Predictive Autonomous Robot Navigation

paper_content:

This paper considers the problem of a robot navigating in a crowded or congested environment. A robot operating in such an environment can get easily blocked by moving humans and other objects. To deal with this problem it is proposed to attempt to predict the motion trajectory of humans and obstacles. Two kinds of prediction are considered: short-term and long-term. The short-term prediction refers to the one-step ahead prediction and the long-term to the prediction of the final destination point of the obstacle's movement. The robot movement is controlled by a partially observable Markov decision process (POMDP). POMDPs are utilized because of their ability to model information about the robot's location and sensory information in a probabilistic manner. The solution of a POMDP is computationally expensive and thus a hierarchical representation of POMDPs is used.
---
paper_title: SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints

paper_content:

This paper addresses the problem of path prediction for multiple interacting agents in a scene, which is a crucial step for many autonomous platforms such as self-driving cars and social robots. We present SoPhie; an interpretable framework based on Generative Adversarial Network (GAN), which leverages two sources of information, the path history of all the agents in a scene, and the scene context information, using images of the scene. To predict a future path for an agent, both physical and social information must be leveraged. Previous work has not been successful to jointly model physical and social interactions. Our approach blends a social attention mechanism with physical attention that helps the model to learn where to look in a large scene and extract the most salient parts of the image relevant to the path. Whereas, the social attention component aggregates information across the different agent interactions and extracts the most important trajectory information from the surrounding neighbors. SoPhie also takes advantage of GAN to generates more realistic samples and to capture the uncertain nature of the future paths by modeling its distribution. All these mechanisms enable our approach to predict socially and physically plausible paths for the agents and to achieve state-of-the-art performance on several different trajectory forecasting benchmarks.
---
paper_title: INFER: INtermediate representations for FuturE pRediction

paper_content:

In urban driving scenarios, forecasting future trajectories of surrounding vehicles is of paramount importance. While several approaches for the problem have been proposed, the best-performing ones tend to require extremely detailed input representations (e.g. image sequences). As a result, such methods do not generalize to datasets they have not been trained on. In this paper, we propose intermediate representations that are particularly well-suited for future prediction. As opposed to using texture (color) information from images, we condition on semantics and train an autoregressive model to accurately predict future trajectories of traffic participants (vehicles) (see Fig. above). We demonstrate that semantics provide a significant boost over techniques that operate over raw pixel intensities/disparities. Uncharacteristic of state-of-the-art approaches, our representations and models generalize across different sensing modalities (stereo imagery, LiDAR, a combination of both), and also across completely different datasets, collected across several cities, and also across countries where people drive on opposite sides of the road (left-handed vs right-handed driving). Additionally, we demonstrate an application of our approach in multi-object tracking (data association). To foster further research in transferable representations and ensure reproducibility, we release all our code and data.33More qualitative and quantitative results, along with code and data can be found at https://rebrand.ly/INFER-results
---
paper_title: Human Trajectory Prediction using Spatially aware Deep Attention Models

paper_content:

Trajectory Prediction of dynamic objects is a widely studied topic in the field of artificial intelligence. Thanks to a large number of applications like predicting abnormal events, navigation system for the blind, etc. there have been many approaches to attempt learning patterns of motion directly from data using a wide variety of techniques ranging from hand-crafted features to sophisticated deep learning models for unsupervised feature learning. All these approaches have been limited by problems like inefficient features in the case of hand crafted features, large error propagation across the predicted trajectory and no information of static artefacts around the dynamic moving objects. We propose an end to end deep learning model to learn the motion patterns of humans using different navigational modes directly from data using the much popular sequence to sequence model coupled with a soft attention mechanism. We also propose a novel approach to model the static artefacts in a scene and using these to predict the dynamic trajectories. The proposed method, tested on trajectories of pedestrians, consistently outperforms previously proposed state of the art approaches on a variety of large scale data sets. We also show how our architecture can be naturally extended to handle multiple modes of movement (say pedestrians, skaters, bikers and buses) simultaneously.
---
paper_title: Multimodal Probabilistic Model-Based Planning for Human-Robot Interaction

paper_content:

This paper presents a method for constructing human-robot interaction policies in settings where multimodality, i.e., the possibility of multiple highly distinct futures, plays a critical role in decision making. We are motivated in this work by the example of traffic weaving, e.g., at highway on-ramps/off-ramps, where entering and exiting cars must swap lanes in a short distance-a challenging negotiation even for experienced drivers due to the inherent multimodal uncertainty of who will pass whom. Our approach is to learn multimodal probability distributions over future human actions from a dataset of human-human exemplars and perform real-time robot policy construction in the resulting environment model through massively parallel sampling of human responses to candidate robot action sequences. Direct learning of these distributions is made possible by recent advances in the theory of conditional variational autoencoders (CVAEs), whereby we learn action distributions simultaneously conditioned on the present interaction history, as well as candidate future robot actions in order to take into account response dynamics. We demonstrate the efficacy of this approach with a human-in-the-loop simulation of a traffic weaving scenario.
---
paper_title: Online maneuver recognition and multimodal trajectory prediction for intersection assistance using non-parametric regression

paper_content:

Maneuver recognition and trajectory prediction of moving vehicles are two important and challenging tasks of advanced driver assistance systems (ADAS) at urban intersections. This paper presents a continuing work to handle these two problems in a consistent framework using non-parametric regression models. We provide a feature normalization scheme and present a strategy for constructing three-dimensional Gaussian process regression models from two-dimensional trajectory patterns. These models can capture spatio-temporal characteristics of traffic situations. Given a new, partially observed and unlabeled trajectory, the maneuver can be recognized online by comparing the likelihoods of the observation data for each individual regression model. Furthermore, we take advantage of our representation for trajectory prediction. Because predicting possible trajectories at urban intersection involves obvious multimodalities and non-linearities, we employ the Monte Carlo method to handle these difficulties. This approach allows the incremental prediction of possible trajectories in situations where unimodal estimators such as Kalman Filters would not work well. The proposed framework is evaluated experimentally in urban intersection scenarios using real-world data.
---
paper_title: Group LSTM: Group Trajectory Prediction in Crowded Scenarios

paper_content:

The analysis of crowded scenes is one of the most challenging scenarios in visual surveillance, and a variety of factors need to be taken into account, such as the structure of the environments, and the presence of mutual occlusions and obstacles. Traditional prediction methods (such as RNN, LSTM, VAE, etc.) focus on anticipating individual’s future path based on the precise motion history of a pedestrian. However, since tracking algorithms are generally not reliable in highly dense scenes, these methods are not easily applicable in real environments. Nevertheless, it is very common that people (friends, couples, family members, etc.) tend to exhibit coherent motion patterns. Motivated by this phenomenon, we propose a novel approach to predict future trajectories in crowded scenes, at the group level. First, by exploiting the motion coherency, we cluster trajectories that have similar motion trends. In this way, pedestrians within the same group can be well segmented. Then, an improved social-LSTM is adopted for future path prediction. We evaluate our approach on standard crowd benchmarks (the UCY dataset and the ETH dataset), demonstrating its efficacy and applicability.
---
paper_title: Generative Multi-Agent Behavioral Cloning

paper_content:

We propose and study the problem of generative multi-agent behavioral cloning, where the goal is to learn a generative, i.e., non-deterministic, multi-agent policy from pre-collected demonstration data. Building upon advances in deep generative models, we present a hierarchical policy framework that can tractably learn complex mappings from input states to distributions over multi-agent action spaces by introducing a hierarchy with macro-intent variables that encode long-term intent. In addition to synthetic settings, we show how to instantiate our framework to effectively model complex interactions between basketball players and generate realistic multi-agent trajectories of basketball gameplay over long time periods. We validate our approach using both quantitative and qualitative evaluations, including a user study comparison conducted with professional sports analysts.
---
paper_title: Multi-Agent Tensor Fusion for Contextual Trajectory Prediction

paper_content:

Accurate prediction of others' trajectories is essential for autonomous driving. Trajectory prediction is challenging because it requires reasoning about agents' past movements, social interactions among varying numbers and kinds of agents, constraints from the scene context, and the stochasticity of human behavior. Our approach models these interactions and constraints jointly within a novel Multi-Agent Tensor Fusion (MATF) network. Specifically, the model encodes multiple agents' past trajectories and the scene context into a Multi-Agent Tensor, then applies convolutional fusion to capture multiagent interactions while retaining the spatial structure of agents and the scene context. The model decodes recurrently to multiple agents' future trajectories, using adversarial loss to learn stochastic predictions. Experiments on both highway driving and pedestrian crowd datasets show that the model achieves state-of-the-art prediction accuracy.
---
paper_title: Pedestrian Path, Pose, and Intention Prediction Through Gaussian Process Dynamical Models and Pedestrian Activity Recognition

paper_content:

According to several reports published by worldwide organizations, thousands of pedestrians die in road accidents every year. Due to this fact, vehicular technologies have been evolving with the intent of reducing these fatalities. This evolution has not finished yet, since, for instance, the predictions of pedestrian paths could improve the current automatic emergency braking systems. For this reason, this paper proposes a method to predict future pedestrian paths, poses, and intentions up to 1 s in advance. This method is based on balanced Gaussian process dynamical models (B-GPDMs), which reduce the 3-D time-related information extracted from key points or joints placed along pedestrian bodies into low-dimensional spaces. The B-GPDM is also capable of inferring future latent positions and reconstruct their associated observations. However, learning a generic model for all kinds of pedestrian activities normally provides less accurate predictions. For this reason, the proposed method obtains multiple models of four types of activity, i.e., walking, stopping, starting, and standing, and selects the most similar model to estimate future pedestrian states. This method detects starting activities 125 ms after the gait initiation with an accuracy of 80% and recognizes stopping intentions 58.33 ms before the event with an accuracy of 70%. Concerning the path prediction, the mean error for stopping activities at a time-to-event (TTE) of 1 s is 238.01 ± 206.93 mm and, for starting actions, the mean error at a TTE of 0 s is 331.93 ± 254.73 mm.
---
paper_title: Long-term Planning by Short-term Prediction

paper_content:

We consider planning problems, that often arise in autonomous driving applications, in which an agent should decide on immediate actions so as to optimize a long term objective. For example, when a car tries to merge in a roundabout it should decide on an immediate acceleration/braking command, while the long term effect of the command is the success/failure of the merge. Such problems are characterized by continuous state and action spaces, and by interaction with multiple agents, whose behavior can be adversarial. We argue that dual versions of the MDP framework (that depend on the value function and the $Q$ function) are problematic for autonomous driving applications due to the non Markovian of the natural state space representation, and due to the continuous state and action spaces. We propose to tackle the planning task by decomposing the problem into two phases: First, we apply supervised learning for predicting the near future based on the present. We require that the predictor will be differentiable with respect to the representation of the present. Second, we model a full trajectory of the agent using a recurrent neural network, where unexplained factors are modeled as (additive) input nodes. This allows us to solve the long-term planning problem using supervised learning techniques and direct optimization over the recurrent neural network. Our approach enables us to learn robust policies by incorporating adversarial elements to the environment.
---
paper_title: Deep Learning Driven Visual Path Prediction from a Single Image

paper_content:

Capabilities of inference and prediction are the significant components of visual systems. Visual path prediction is an important and challenging task among them, with the goal to infer the future path of a visual object in a static scene. This task is complicated as it needs high-level semantic understandings of both the scenes and underlying motion patterns in video sequences. In practice, cluttered situations have also raised higher demands on the effectiveness and robustness of models. Motivated by these observations, we propose a deep learning framework, which simultaneously performs deep feature learning for visual representation in conjunction with spatiotemporal context modeling. After that, a unified path-planning scheme is proposed to make accurate path prediction based on the analytic results returned by the deep context models. The highly effective visual representation and deep context models ensure that our framework makes a deep semantic understanding of the scenes and motion patterns, consequently improving the performance on visual path prediction task. In experiments, we extensively evaluate the model’s performance by constructing two large benchmark datasets from the adaptation of video tracking datasets. The qualitative and quantitative experimental results show that our approach outperforms the state-of-the-art approaches and owns a better generalization capability.
---
paper_title: Pedestrian path prediction using body language traits

paper_content:

Driver Assistance Systems have achieved a high level of maturity in the latest years. As an example of that, sophisticated pedestrian protection systems are already available in a number of commercial vehicles from several OEMs. However, accurate pedestrian path prediction is needed in order to go a step further in terms of safety and reliability, since it can make the difference between effective and non-effective intervention. In this paper, we consider the three-dimensional pedestrian body language in order to perform path prediction in a probabilistic framework. For this purpose, the different body parts and joints are detected using stereo vision. We propose the use of GPDM (Gaussian Process Dynamical Models) for reducing the high dimensionality of the input feature vector (composed by joints and displacement vectors) in the 3D pose space and for learning the pedestrian dynamics in a latent space. Experimental results show that accurate path prediction can be achieved at a time horizon of ≈ 0.8 s.
---
paper_title: Social LSTM: Human Trajectory Prediction in Crowded Spaces

paper_content:

Pedestrians follow different trajectories to avoid obstacles and accommodate fellow pedestrians. Any autonomous vehicle navigating such a scene should be able to foresee the future positions of pedestrians and accordingly adjust its path to avoid collisions. This problem of trajectory prediction can be viewed as a sequence generation task, where we are interested in predicting the future trajectory of people based on their past positions. Following the recent success of Recurrent Neural Network (RNN) models for sequence prediction tasks, we propose an LSTM model which can learn general human movement and predict their future trajectories. This is in contrast to traditional approaches which use hand-crafted functions such as Social forces. We demonstrate the performance of our method on several public datasets. Our model outperforms state-of-the-art methods on some of these datasets. We also analyze the trajectories predicted by our model to demonstrate the motion behaviour learned by our model.
---
paper_title: Threat assessment design for driver assistance system at intersections

paper_content:

This paper considers the decision-making problem for a human-driven vehicle crossing a road intersection in the presence of other, potentially errant, drivers. Our approach relies on a novel threat assessment module, which combines an intention predictor based on support vector machines with an efficient threat assessor using rapidly-exploring random trees. This module warns the host driver with the computed threat level and the corresponding best “escape maneuver” through the intersection, if the threat is sufficiently large. Through experimental results with small autonomous and human-driven vehicles, we demonstrate that this threat assessment module can be used in real-time to minimize the risk of collision.
---
paper_title: SS-LSTM: A Hierarchical LSTM Model for Pedestrian Trajectory Prediction

paper_content:

Pedestrian trajectory prediction is an extremely challenging problem because of the crowdedness and clutter of the scenes. Previous deep learning LSTM-based approaches focus on the neighbourhood influence of pedestrians but ignore the scene layouts in pedestrian trajectory prediction. In this paper, a novel hierarchical LSTM-based network is proposed to consider both the influence of social neighbourhood and scene layouts. Our SS-LSTM, which stands for Social-Scene-LSTM, uses three different LSTMs to capture person, social and scene scale information. We also use a circular shape neighbourhood setting instead of the traditional rectangular shape neighbourhood in the social scale. We evaluate our proposed method against two baseline methods and a state-of-art technique on three public datasets. The results show that our method outperforms other methods and that using circular shape neighbourhood improves the prediction accuracy.
---
paper_title: Social Attention: Modeling Attention in Human Crowds

paper_content:

Robots that navigate through human crowds need to be able to plan safe, efficient, and human predictable trajectories. This is a particularly challenging problem as it requires the robot to predict future human trajectories within a crowd where everyone implicitly cooperates with each other to avoid collisions. Previous approaches to human trajectory prediction have modeled the interactions between humans as a function of proximity. However, that is not necessarily true as some people in our immediate vicinity moving in the same direction might not be as important as other people that are further away, but that might collide with us in the future. In this work, we propose Social Attention, a novel trajectory prediction model that captures the relative importance of each person when navigating in the crowd, irrespective of their proximity. We demonstrate the performance of our method against a state-of-the-art approach on two publicly available crowd datasets and analyze the trained attention model to gain a better understanding of which surrounding agents humans attend to, when navigating in a crowd.
---
paper_title: MX-LSTM: Mixing Tracklets and Vislets to Jointly Forecast Trajectories and Head Poses

paper_content:

Recent approaches on trajectory forecasting use tracklets to predict the future positions of pedestrians exploiting Long Short Term Memory (LSTM) architectures. This paper shows that adding vislets, that is, short sequences of head pose estimations, allows to increase significantly the trajectory forecasting performance. We then propose to use vislets in a novel framework called MX-LSTM, capturing the interplay between tracklets and vislets thanks to a joint unconstrained optimization of full covariance matrices during the LSTM backpropagation. At the same time, MX-LSTM predicts the future head poses, increasing the standard capabilities of the long-term trajectory forecasting approaches. With standard head pose estimators and an attentional-based social pooling, MX-LSTM scores the new trajectory forecasting state-of-the-art in all the considered datasets (Zara01, Zara02, UCY, and TownCentre) with a dramatic margin when the pedestrians slow down, a case where most of the forecasting approaches struggle to provide an accurate solution.
---
paper_title: A Data-driven Model for Interaction-Aware Pedestrian Motion Prediction in Object Cluttered Environments

paper_content:

This paper reports on a data-driven, interaction-aware motion prediction approach for pedestrians in environments cluttered with static obstacles. When navigating in such workspaces shared with humans, robots need accurate motion predictions of the surrounding pedestrians. Human navigation behavior is mostly influenced by their surrounding pedestrians and by the static obstacles in their vicinity. In this paper we introduce a new model based on Long-Short Term Memory (LSTM) neural networks, which is able to learn human motion behavior from demonstrated data. To the best of our knowledge, this is the first approach using LSTMs, that incorporates both static obstacles and surrounding pedestrians for trajectory forecasting. As part of the model, we introduce a new way of encoding surrounding pedestrians based on a 1d-grid in polar angle space. We evaluate the benefit of interaction-aware motion prediction and the added value of incorporating static obstacles on both simulation and real-world datasets by comparing with state-of-the-art approaches. The results show, that our new approach outperforms the other approaches while being very computationally efficient and that taking into account static obstacles for motion predictions significantly improves the prediction accuracy, especially in cluttered environments.
---
paper_title: Multi-Modal Trajectory Prediction of Surrounding Vehicles with Maneuver based LSTMs

paper_content:

To safely and efficiently navigate through complex traffic scenarios, autonomous vehicles need to have the ability to predict the future motion of surrounding vehicles. Multiple interacting agents, the multi-modal nature of driver behavior, and the inherent uncertainty involved in the task make motion prediction of surrounding vehicles a challenging problem. In this paper, we present an LSTM model for interaction aware motion prediction of surrounding vehicles on freeways. Our model assigns confidence values to maneuvers being performed by vehicles and outputs a multi-modal distribution over future motion based on them. We compare our approach with the prior art for vehicle motion prediction on the publicly available NGSIM US-101 and I-80 datasets. Our results show an improvement in terms of RMS values of prediction error. We also present an ablative analysis of the components of our proposed model and analyze the predictions made by the model in complex traffic scenarios.
---
paper_title: Unfreezing the robot: Navigation in dense, interacting crowds

paper_content:

In this paper, we study the safe navigation of a mobile robot through crowds of dynamic agents with uncertain trajectories. Existing algorithms suffer from the “freezing robot” problem: once the environment surpasses a certain level of complexity, the planner decides that all forward paths are unsafe, and the robot freezes in place (or performs unnecessary maneuvers) to avoid collisions. Since a feasible path typically exists, this behavior is suboptimal. Existing approaches have focused on reducing the predictive uncertainty for individual agents by employing more informed models or heuristically limiting the predictive covariance to prevent this overcautious behavior. In this work, we demonstrate that both the individual prediction and the predictive uncertainty have little to do with the frozen robot problem. Our key insight is that dynamic agents solve the frozen robot problem by engaging in “joint collision avoidance”: They cooperatively make room to create feasible trajectories. We develop IGP, a nonparametric statistical model based on dependent output Gaussian processes that can estimate crowd interaction from data. Our model naturally captures the non-Markov nature of agent trajectories, as well as their goal-driven navigation. We then show how planning in this model can be efficiently implemented using particle based inference. Lastly, we evaluate our model on a dataset of pedestrians entering and leaving a building, first comparing the model with actual pedestrians, and find that the algorithm either outperforms human pedestrians or performs very similarly to the pedestrians. We also present an experiment where a covariance reduction method results in highly overcautious behavior, while our model performs desirably.
---
paper_title: Visual Path Prediction in Complex Scenes with Crowded Moving Objects

paper_content:

This paper proposes a novel path prediction algorithm for progressing one step further than the existing works focusing on single target path prediction. In this paper, we consider moving dynamics of co-occurring objects for path prediction in a scene that includes crowded moving objects. To solve this problem, we first suggest a two-layered probabilistic model to find major movement patterns and their cooccurrence tendency. By utilizing the unsupervised learning results from the model, we present an algorithm to find the future location of any target object. Through extensive qualitative/quantitative experiments, we show that our algorithm can find a plausible future path in complex scenes with a large number of moving objects.
---
paper_title: Using EM to learn motion behaviors of persons with mobile robots

paper_content:

We propose a method for learning models of people's motion behaviors in indoor environments. As people move through their environments, they do not move randomly. Instead, they often engage in typical motion patterns, related to specific locations that they might be interested in approaching and specific trajectories that they might follow in doing so. Knowledge about such patterns may enable a mobile robot to develop improved people following and obstacle avoidance skills. This paper proposes an algorithm that learns collections of typical trajectories that characterize a person's motion patterns. Data, recorded by mobile robots equipped with laser-range finders, is clustered into different types of motion using the popular expectation maximization algorithm, while simultaneously learning multiple motion patterns. Experimental results, obtained using data collected in a domestic residence and in an office building, illustrate that highly predictive models of human motion patterns can be learned.
---
paper_title: Long-term vehicle motion prediction

paper_content:

Future driver assistance systems will have to cope with complex traffic situations, especially in the road crossing scenario. To detect potentially hazardous situations as early as possible, it is therefore desirable to know the position and motion of the ego-vehicle and vehicles around it for several seconds in advance. For this purpose, we propose in this study a long-term prediction approach based on a combined trajectory classification and particle filter framework. As a measure for the similarity between trajectories, we introduce the quaternion-based rotationally invariant longest common subsequence (QRLCS) metric. The trajectories are classified by a radial basis function (RBF) classifier with an architecture that is able to process trajectories of arbitrary non-uniform length. The particle filter framework simultaneously tracks and assesses a large number of motion hypotheses (∼102), where the class-specific probabilities estimated by the RBF classifier are used as a-priori probabilities for the hypotheses of the particle filter. The hypotheses are clustered with a mean-shift technique and are assigned a likelihood value. Motion prediction is performed based on the cluster centre with the highest likelihood. While traditional motion prediction based on curve radius and acceleration is inaccurate especially during turning manoeuvres, we show that our approach achieves a reasonable motion prediction even for long prediction intervals of 3 s for these complex motion patterns.
---
paper_title: Unsupervised robot learning to predict person motion

paper_content:

Socially interacting robots will need to understand the intentions and recognize the behaviors of people they come in contact with. In this paper we look at how a robot can learn to recognize and predict people's intended path based on its own observations of people over time. Our approach uses people tracking on the robot from either RGBD cameras or LIDAR. The tracks are separated into homogeneous motion classes using a pre-trained SVM. Then the individual classes are clustered and prototypes are extracted from each cluster. These are then used to predict a person's future motion based on matching to a partial prototype and using the rest of the prototype as the predicted motion. Results from experiments in a kitchen environment in our lab demonstrate the capabilities of the proposed method.
---
paper_title: Predicting trajectories of vehicles using large-scale motion priors

paper_content:

We present a simple yet effective paradigm to accurately predict the future trajectories of observed vehicles in dense city environments. We equipped a large fleet of cars with cameras and performed city-scale structure-from-motion to accurately reconstruct 10M positions of their trajectories spanning over 1000h of driving.We demonstrate that this information can be used as a powerful high-fidelity prior to predict future trajectories of newly observed vehicles in the area without the need for any knowledge of road infrastructure or vehicle motion models. By relating the current position of the observed car to a large dataset of the previously exhibited motion in the area we can directly perform prediction of its future position.We evaluate our method on two large-scale data sets from San Francisco and New York City and demonstrate an order of magnitude improvement compared to a linear-motion based method. We also demonstrate that the performance naturally improves with the amount of data and ultimately yields a system that can accurately predict vehicle motion in challenging situations across extremes in traffic, time, and weather conditions.
---
paper_title: Recognition of situation classes at road intersections

paper_content:

The recognition and prediction of situations is an indispensable skill of future driver assistance systems. This study focuses on the recognition of situations involving two vehicles at intersections. For each vehicle, a set of possible future motion trajectories is estimated and rated based on a motion database for a time interval of 2–4 seconds ahead. Realistic situations are generated by a pairwise combination of these individual motion trajectories and classified according to nine categories with a polynomial classifier. In the proposed framework, situations are penalised for which the time to collision significantly exceeds the typical human reaction time. The correspondingly favoured situations are combined by a probabilistic framework, resulting in a more reliable situation recognition and collision detection than obtained based on independent motion hypotheses. The proposed method is evaluated on a real-world differential GPS data set acquired during a test drive of 10 km, including three road intersections. Our method is typically able to recognise the situation correctly about 1–2 seconds before the distance to the intersection centre becomes minimal.
---
paper_title: Will the Pedestrian Cross? A Study on Pedestrian Path Prediction

paper_content:

Future vehicle systems for active pedestrian safety will not only require a high recognition performance but also an accurate analysis of the developing traffic situation. In this paper, we present a study on pedestrian path prediction and action classification at short subsecond time intervals. We consider four representative approaches: two novel approaches (based on Gaussian process dynamical models and probabilistic hierarchical trajectory matching) that use augmented features derived from dense optical flow and two approaches as baseline that use positional information only (a Kalman filter and its extension to interacting multiple models). In experiments using stereo vision data obtained from a vehicle, we investigate the accuracy of path prediction and action classification at various time horizons, the effect of various errors (image localization, vehicle egomotion estimation), and the benefit of the proposed approaches. The scenario of interest is that of a crossing pedestrian, who might stop or continue walking at the road curbside. Results indicate similar performance of the four approaches on walking motion, with near-linear dynamics. During stopping, however, the two newly proposed approaches, with nonlinear and/or higher order models and augmented motion features, achieve a more accurate position prediction of 10-50 cm at a time horizon of 0-0.77 s around the stopping event.
---
paper_title: Motion Prediction of Traffic Actors for Autonomous Driving using Deep Convolutional Networks.

paper_content:

Recent algorithmic improvements and hardware breakthroughs resulted in a number of success stories in the field of AI impacting our daily lives. However, despite its ubiquity AI is only just starting to make advances in what may arguably have the largest impact thus far, the nascent field of autonomous driving. In this work we discuss this important topic and address one of crucial aspects of the emerging area, the problem of predicting future state of autonomous vehicle's surrounding necessary for safe and efficient operations. We introduce a deep learning-based approach that takes into account current state of traffic actors and produces rasterized representations of each actor's vicinity. The raster images are then used by deep convolutional models to infer future movement of actors while accounting for inherent uncertainty of the prediction task. Extensive experiments on real-world data strongly suggest benefits of the proposed approach. Moreover, following successful tests the system was deployed to a fleet of autonomous vehicles.
---
paper_title: Modelling Smooth Paths Using Gaussian Processes

paper_content:

A generative model based on the gaussian mixture model and gaussian processes is presented in this paper. Typical motion paths are learnt and then used for motion prediction using this model. The principal novel aspect of this approach is the modelling of paths using gaussian processes. It allows the representation of smooth trajectories and avoids discretization problems found in most existing methods. Gaussian processes not only provides a comprehensive and formal theoretical framework to work with, it also lends itself naturally to path clustering using gaussian mixture models. Learning is performed using expectation maximization where the E-Step uses variational methods to maximize its lower bound before optimization over parameters are performed in the M-Step.
---
paper_title: GLMP- realtime pedestrian path prediction using global and local movement patterns

paper_content:

We present a novel real-time algorithm to predict the path of pedestrians in cluttered environments. Our approach makes no assumption about pedestrian motion or crowd density, and is useful for short-term as well as long-term prediction. We interactively learn the characteristics of pedestrian motion and movement patterns from 2D trajectories using Bayesian inference. These include local movement patterns corresponding to the current and preferred velocities and global characteristics such as entry points and movement features. Our approach involves no precomputation and we demonstrate the real-time performance of our prediction algorithm on sparse and noisy trajectory data extracted from dense indoor and outdoor crowd videos. The combination of local and global movement patterns can improve the accuracy of long-term prediction by 12–18% over prior methods in high-density videos.
---
paper_title: Learning Motion Patterns of People for Compliant Robot Motion

paper_content:

Whenever people move through their environments they do not move randomly. Instead, they usually follow specific trajectories or motion patterns corresponding to their intentions. Knowledge about such patterns enables a mobile robot to robustly keep track of persons in its environment and to improve its behavior. In this paper we propose a technique for learning collections of trajectories that characterize typical motion patterns of persons. Data recorded with laser-range finders are clustered using the expectation maximization algorithm. Based on the result of the clustering process, we derive a hidden Markov model that is applied to estimate the current and future positions of persons based on sensory input. We also describe how to incorporate the probabilistic belief about the potential trajectories of persons into the path planning process of a mobile robot. We present several experiments carried out in different environments with a mobile robot equipped with a laser-range scanner and a camera system...
---
paper_title: Pedestrian Behavior Prediction based on Motion Patterns for Vehicle-to-Pedestrian Collision Avoidance

paper_content:

This paper proposes a prediction method for vehicle-to-pedestrian collision avoidance, which learns and then predicts pedestrian behaviors as their motion instances are being observed. During learning, known trajectories are clustered to form motion patterns (MP), which become knowledge a priori to a multi-level prediction model that predicts long-term or short-term pedestrian behaviors. Simulation results show that it works well in a complex structured environment and the prediction is consistent with actual behaviors.
---
paper_title: Robot navigation in dense human crowds: the case for cooperation

paper_content:

We consider mobile robot navigation in dense human crowds. In particular, we explore two questions. Can we design a navigation algorithm that encourages humans to cooperate with a robot? Would such cooperation improve navigation performance? We address the first question by developing a probabilistic predictive model of cooperative collision avoidance and goal-oriented behavior by extending the interacting Gaussian processes approach to include multiple goals and stochastic movement duration. We answer the second question with an extensive quantitative study of robot navigation in dense human crowds (488 runs completed), specifically testing how cooperation models effect navigation performance. We find that the “multiple goal” interacting Gaussian processes algorithm performs comparably with human teleoperators in crowd densities near 1 person/m2, while a state of the art noncooperative planner exhibits unsafe behavior more than 3 times as often as this multiple goal extension, and more than twice as often as the basic interacting Gaussian processes. Furthermore, a reactive planner based on the widely used “dynamic window” approach fails for crowd densities above 0.55 people/m2. Based on these experimental results, and previous theoretical observations, we conclude that a cooperation model is important for safe and efficient robot navigation in dense human crowds.
---
paper_title: Socially-aware robot navigation: A learning approach

paper_content:

The ability to act in a socially-aware way is a key skill for robots that share a space with humans. In this paper we address the problem of socially-aware navigation among people that meets objective criteria such as travel time or path length as well as subjective criteria such as social comfort. Opposed to model-based approaches typically taken in related work, we pose the problem as an unsupervised learning problem. We learn a set of dynamic motion prototypes from observations of relative motion behavior of humans found in publicly available surveillance data sets. The learned motion prototypes are then used to compute dynamic cost maps for path planning using an any-angle A* algorithm. In the evaluation we demonstrate that the learned behaviors are better in reproducing human relative motion in both criteria than a Proxemics-based baseline method.
---
paper_title: Learning Collective Crowd Behaviors with Dynamic Pedestrian-Agents

paper_content:

Collective behaviors characterize the intrinsic dynamics of the crowds. Automatically understanding collective crowd behaviors has important applications to video surveillance, traffic management and crowd control, while it is closely related to scientific fields such as statistical physics and biology. In this paper, a new mixture model of dynamic pedestrian-Agents (MDA) is proposed to learn the collective behavior patterns of pedestrians in crowded scenes from video sequences. From agent-based modeling, each pedestrian in the crowd is driven by a dynamic pedestrian-agent, which is a linear dynamic system with initial and termination states reflecting the pedestrian's belief of the starting point and the destination. The whole crowd is then modeled as a mixture of dynamic pedestrian-agents. Once the model parameters are learned from the trajectories extracted from videos, MDA can simulate the crowd behaviors. It can also infer the past behaviors and predict the future behaviors of pedestrians given their partially observed trajectories, and classify them different pedestrian behaviors. The effectiveness of MDA and its applications are demonstrated by qualitative and quantitative experiments on various video surveillance sequences.
---
paper_title: Gaussian process regression flow for analysis of motion trajectories

paper_content:

Recognition of motions and activities of objects in videos requires effective representations for analysis and matching of motion trajectories. In this paper, we introduce a new representation specifically aimed at matching motion trajectories. We model a trajectory as a continuous dense flow field from a sparse set of vector sequences using Gaussian Process Regression. Furthermore, we introduce a random sampling strategy for learning stable classes of motions from limited data. Our representation allows for incrementally predicting possible paths and detecting anomalous events from online trajectories. This representation also supports matching of complex motions with acceleration changes and pauses or stops within a trajectory. We use the proposed approach for classifying and predicting motion trajectories in traffic monitoring domains and test on several data sets. We show that our approach works well on various types of complete and incomplete trajectories from a variety of video data sets with different frame rates.
---
paper_title: A fast marching level set method for monotonically advancing fronts.

paper_content:

Abstract ::: A fast marching level set method is presented for monotonically advancing fronts, which leads to an extremely fast scheme for solving the Eikonal equation. Level set methods are numerical techniques for computing the position of propagating fronts. They rely on an initial value partial differential equation for a propagating level set function and use techniques borrowed from hyperbolic conservation laws. Topological changes, corner and cusp development, and accurate determination of geometric properties such as curvature and normal direction are naturally obtained in this setting. This paper describes a particular case of such methods for interfaces whose speed depends only on local position. The technique works by coupling work on entropy conditions for interface motion, the theory of viscosity solutions for Hamilton-Jacobi equations, and fast adaptive narrow band level set methods. The technique is applicable to a variety of problems, including shape-from-shading problems, lithographic development calculations in microchip manufacturing, and arrival time problems in control theory.
---
paper_title: Search-based path planning with homotopy class constraints

paper_content:

Homotopy classes of trajectories, arising due to the presence of obstacles, are defined as sets of trajectories that can be transformed into each other by gradual bending and stretching without colliding with obstacles. The problem of exploring/finding the different homotopy classes in an environment and the problem of finding least-cost paths restricted to a specific homotopy class (or not belonging to certain homotopy classes) arises frequently in such applications as predicting paths for unpredictable entities and deployment of multiple agents for efficient exploration of an environment. In (Bhattacharya, Kumar, and Likhachev 2010) we have shown how homotopy classes of trajectories on a two-dimensional plane with obstacles can be classified and identified using the Cauchy Integral Theorem and the Residue Theorem from Complex Analysis. In more recent work (Bhattacharya, Likhachev, and Kumar 2011) we extended this representation to three-dimensional spaces by exploiting certain laws from the Theory of Electromagnetism (Biot-Savart law and Ampere's Law) for representing and identifying homotopy classes in three dimensions in an efficient way. Using such a representation, we showed that homotopy class constraints can be seamlessly weaved into graph search techniques for determining optimal path constrained to certain homotopy classes or forbidden from others, as well as for exploring different homotopy classes in an environment.
---
paper_title: Natural vision based method for predicting pedestrian behaviour in urban environments

paper_content:

This paper proposes to model pedestrian behaviour in urban scenes by combining the principles of urban planning and the sociological concept of Natural Vision. This model assumes that the environment perceived by pedestrians is composed of multiple potential fields that influence their behaviour. These fields are derived from static scene elements like side-walks, cross-walks, buildings, shops entrances and dynamic obstacles like cars and buses for instance. Using this model, autonomous cars increase their level of situational awareness in the local urban space, with the ability to infer probable pedestrian paths in the scene to predict, for example, legal and illegal crossings.
---
paper_title: Intent-aware long-term prediction of pedestrian motion

paper_content:

We present a method to predict long-term motion of pedestrians, modeling their behavior as jump-Markov processes with their goal a hidden variable. Assuming approximately rational behavior, and incorporating environmental constraints and biases, including time-varying ones imposed by traffic lights, we model intent as a policy in a Markov decision process framework. We infer pedestrian state using a Rao-Blackwellized filter, and intent by planning according to a stochastic policy, reflecting individual preferences in aiming at the same goal.
---
paper_title: Bayesian intention inference for trajectory prediction with an unknown goal destination

paper_content:

Contextual cues can provide a rich source of information for robots that operate in the presence of other agents such as people, animals, vehicles and fellow robots. We are interested in context, in the form of the behavioural intent of an agent, for enhanced trajectory prediction. We present a Bayesian framework that estimates both the intended goal destination and future trajectory of a mobile agent moving among multiple static obstacles. Our method is based on multi-modal hypotheses of the intended goal, and is focused primarily on the long-term trajectory of the agent. We propose a computationally efficient solution and demonstrate its behaviour in a pedestrian scenario with a real-world data set. Results show the benefits of our method in comparison to traditional trajectory prediction methods and illustrate the feasibility of integration with higher-level planning algorithms.
---
paper_title: Pedestrian's Trajectory Forecast in Public Traffic with Artificial Neural Networks

paper_content:

This paper focuses on forecasting of pedestrian's short-time trajectories up to 2.5 s for traffic safety applications. We present a self-learning approach based on artificial neural network movement models and compare it to traditional constant velocity Kalman Filter prediction and extrapolation of polynomials fitted using a least-squares error. Trajectories of uninstructed pedestrians in public traffic at a real urban intersection are acquired by a wide angle stereo camera setup in combination with a 3D head tracking framework. Results using this real-world data show that the artificial neural network significantly improves forecast quality compared to other approaches especially for critical traffic scenes including velocity changes such as starting and stopping. For those velocity changes a reduction of position estimation errors of about 21% compared to the Kalman Filter and to extrapolation of polynomials is obtained. By means of a concrete pedestrian-vehicle scenario we demonstrate the benefit of the proposed approach for an advanced driver assistant system in terms of reaction time.
---
paper_title: Pedestrian Behavior Modeling From Stationary Crowds With Applications to Intelligent Surveillance

paper_content:

Pedestrian behavior modeling and analysis is important for crowd scene understanding and has various applications in video surveillance. Stationary crowd groups are a key factor influencing pedestrian walking patterns but was mostly ignored in the literature. It plays different roles for different pedestrians in a crowded scene and can change over time. In this paper, a novel model is proposed to model pedestrian behaviors by incorporating stationary crowd groups as a key component. Through inference on the interactions between stationary crowd groups and pedestrians, our model can be used to investigate pedestrian behaviors. The effectiveness of the proposed model is demonstrated through multiple applications, including walking path prediction, destination prediction, personality attribute classification, and abnormal event detection. To evaluate our model, two large pedestrian walking route datasets are built. The walking routes of around 15 000 pedestrians from two crowd surveillance videos are manually annotated. The datasets will be released to the public and benefit future research on pedestrian behavior analysis and crowd scene understanding.
---
paper_title: Novel planning-based algorithms for human motion prediction

paper_content:

Human motion prediction from visual tracking is a challenging problem with a wide array of applications such as robotics, video surveillance and situation understanding. Recently, planning-based methods -which assume that people move by planning over a cost function- have emerged as one of the most promising alternatives. Nevertheless, state of the art planning based algorithms have shortcomings regarding their computational complexity and ability to predict for arbitrary time intervals. This paper addresses these shortcomings by leveraging alternative planning techniques (Fast Marching Method) and formulating efficient algorithms for goal estimation and full spatiotemporal prediction with lower complexity than comparable approaches. In preliminary experiments, the proposed method significantly outperforms the accuracy of the current state-of-the-art approach while reducing the computation time by a factor of 30 using a parallel version of our algorithm.
---
paper_title: Goal-directed pedestrian model for long-term motion prediction with application to robot motion planning

paper_content:

A probabilistic goal-directed model is proposed for pedestrian motion using navigation function and statistics of human motion gathered in the environment. In comparison with existing models, this model is both computationally inexpensive and does not fail when the optimal direction of motion in terms of this model is non-unique. We further introduce a Rapidly-Exploring Random Tree (RRT)-based path planner developed for planning in state-time space. With the help of an improved distance metric, the planner is much faster than RRT-Blossom [11] in complex maps. In an environment with 10 pedestrians, the planner and motion prediction combined can perform in near-real time.
---
paper_title: Inferring "Dark Matter" and "Dark Energy" from Videos

paper_content:

This paper presents an approach to localizing functional objects in surveillance videos without domain knowledge about semantic object classes that may appear in the scene. Functional objects do not have discriminative appearance and shape, but they affect behavior of people in the scene. For example, they "attract" people to approach them for satisfying certain needs (e.g., vending machines could quench thirst), or "repel" people to avoid them (e.g., grass lawns). Therefore, functional objects can be viewed as "dark matter", emanating "dark energy" that affects people's trajectories in the video. To detect "dark matter" and infer their "dark energy" field, we extend the Lagrangian mechanics. People are treated as particle-agents with latent intents to approach "dark matter" and thus satisfy their needs, where their motions are subject to a composite "dark energy" field of all functional objects in the scene. We make the assumption that people take globally optimal paths toward the intended "dark matter" while avoiding latent obstacles. A Bayesian framework is used to probabilistically model: people's trajectories and intents, constraint map of the scene, and locations of functional objects. A data-driven Markov Chain Monte Carlo (MCMC) process is used for inference. Our evaluation on videos of public squares and courtyards demonstrates our effectiveness in localizing functional objects and predicting people's trajectories in unobserved parts of the video footage.
---
paper_title: Timed-Elastic-Bands for time-optimal point-to-point nonlinear model predictive control

paper_content:

This contribution presents a novel approach for nonlinear time-optimal model predictive control (MPC) based on Timed-Elastic-Bands (TEB). The TEB merges the states, control inputs and time intervals into a joint trajectory representation which enables planning of time-optimal trajectories in the context of model predictive control. Model predictive control integrates the planning of the optimal trajectory with state feedback in the control loop. The TEB approach formulates the fixed horizon optimal control problem for point-to-point transitions as a nonlinear program. The comparative analysis of the TEB approach with state-of-the-art approaches demonstrates its computational efficiency. The TEB approach generates a trajectory that approximates the analytical time-optimal trajectory in few iterations. This efficiency enables the refinement of the planned state and control sequence within the underlying closed-loop control during runtime.
---
paper_title: Exploration of Pedestrian Gap-Acceptance Behavior at Selected Locations

paper_content:

This paper describes the efforts to evaluate pedestrian gap acceptance as part of a recent TCRP-NCHRP project. Pedestrian crossing data were collected at 42 study sites in seven states. From those sites, 45 pedestrian approaches had at least one crossing event where a pedestrian rejected at least one gap, and 11 of those approaches had at least 20 such crossing events. Focusing on the 11 approaches, researchers evaluated the gap-acceptance behavior of crossing pedestrians with a two-part analysis: behavioral analysis and statistical analysis. Behavioral analysis revealed that pedestrians did not always wait to cross the street when all lanes were completely clear; instead, they anticipated that the lanes would clear as they crossed and used a "rolling gap" to cross the street. Statistical analysis revealed that the 11 approaches had 85th percentile accepted gaps between 5.3 and 9.4 s, with a trend of increasing gap length as crossing distance increased. All the observed 85th percentile accepted gaps were ...
---
paper_title: Intention-Aware Motion Planning

paper_content:

As robots venture into new application domains as autonomous vehicles on the road or as domestic helpers at home, they must recognize human intentions and behaviors in order to operate effectively. This paper investigates a new class of motion planning problems with uncertainty in human intention. We propose a method for constructing a practical model by assuming a finite set of unknown intentions. We first construct a motion model for each intention in the set and then combine these models together into a single Mixed Observability Markov Decision Process (MOMDP), which is a structured variant of the more common Partially Observable Markov Decision Process (POMDP). By leveraging the latest advances in POMDP/MOMDP approximation algorithms, we can construct and solve moderately complex models for interesting robotic tasks. Experiments in simulation and with an autonomous vehicle show that the proposed method outperforms common alternatives because of its ability in recognizing intentions and using the information effectively for decision making.
---
paper_title: Planning-based prediction for pedestrians

paper_content:

We present a novel approach for determining robot movements that efficiently accomplish the robot's tasks while not hindering the movements of people within the environment. Our approach models the goal-directed trajectories of pedestrians using maximum entropy inverse optimal control. The advantage of this modeling approach is the generality of its learned cost function to changes in the environment and to entirely different environments. We employ the predictions of this model of pedestrian trajectories in a novel incremental planner and quantitatively show the improvement in hindrance-sensitive robot trajectory planning provided by our approach.
---
paper_title: Human Motion Prediction Under Social Grouping Constraints

paper_content:

Accurate long-term prediction of human motion in populated spaces is an important but difficult task for mobile robots and intelligent vehicles. What makes this task challenging is that human motion is influenced by a large variety of factors including the person's intention, the presence, attributes, actions, social relations and social norms of other surrounding agents, and the geometry and semantics of the environment. In this paper, we consider the problem of computing human motion predictions that account for such factors. We formulate the task as an MDP planning problem with stochastic policies and propose a weighted random walk algorithm in which each agent is locally influenced by social forces from other nearby agents. The novelty of this paper is that we incorporate social grouping information into the prediction process reflecting the soft formation constraints that groups typically impose to their members' motion. We show that our method makes more accurate predictions than three state-of-the-art methods in terms of probabilistic and geometrical performance metrics.
---
paper_title: Interactive navigation of multiple agents in crowded environments

paper_content:

We present a novel approach for interactive navigation and planning of multiple agents in crowded scenes with moving obstacles. Our formulation uses a precomputed roadmap that provides macroscopic, global connectivity for wayfinding and combines it with fast and localized navigation for each agent. At runtime, each agent senses the environment independently and computes a collision-free path based on an extended "Velocity Obstacles" concept. Furthermore, our algorithm ensures that each agent exhibits no oscillatory behaviors. We have tested the performance of our algorithm in several challenging scenarios with a high density of virtual agents. In practice, the algorithm performance scales almost linearly with the number of agents and can run at interactive rates on multi-core processors.
---
paper_title: Intent-aware long-term prediction of pedestrian motion

paper_content:

We present a method to predict long-term motion of pedestrians, modeling their behavior as jump-Markov processes with their goal a hidden variable. Assuming approximately rational behavior, and incorporating environmental constraints and biases, including time-varying ones imposed by traffic lights, we model intent as a policy in a Markov decision process framework. We infer pedestrian state using a Rao-Blackwellized filter, and intent by planning according to a stochastic policy, reflecting individual preferences in aiming at the same goal.
---
paper_title: Long-term Path Prediction in Urban Scenarios using Circular Distributions

paper_content:

Abstract Human ability to foresee the near future plays a key role in everyone's life to prevent potentially dangerous situations. To be able to make predictions is crucial when people have to interact with the surrounding environment. Modeling such capability can lead to the design of automated warning systems and provide moving robots with an intelligent way of interaction with changing situation. In this work we focus on a typical urban human-scene where we aim at predicting an agent's behavior using a stochastic model. In this approach, we fuse the various factors that would contribute to a human motion in different contexts. Our method uses previously observed trajectories to build point-wise circular distributions that after combination, provide a statistical smooth prediction towards the most likely areas. More specifically, a ray-launching procedure, based on a semantic segmentation, gives a coarse scene representation for collision avoidance; a nearly-constant velocity dynamic model smooths the acceleration progression and knowledge of the agent's destination may further steer the path prediction. Experimental results in structured scenes, validate the effectiveness of the method in predicting paths in comparison to actual trajectories.
---
paper_title: A Game-Theoretic Approach to Replanning-Aware Interactive Scene Prediction and Planning

paper_content:

This paper presents a novel cooperative-driving prediction and planning framework for dynamic environments based on the methods of game theory. The proposed algorithm can be used for highly automated driving on highways or as a sophisticated prediction module for advanced driver-assistance systems with no need for intervehicle communication. The main contribution of this paper is a model-based interaction-aware motion prediction of all vehicles in a scene. In contrast to other state-of-the-art approaches, the system also models the replanning capabilities of all drivers. With that, the driving strategy is able to capture complex interactions between vehicles, thus planning maneuver sequences over longer time horizons. It also enables an accurate prediction of traffic for the next immediate time step. The prediction model is supported by an interpretation of what other drivers intend to do, how they interact with traffic, and the ongoing observation. As part of the prediction loop, the proposed planning strategy incorporates the expected reactions of all traffic participants, offering cooperative and robust driving decisions. By means of experimental results under simulated highway scenarios, the validity of the proposed concept and its real-time capability is demonstrated.
---
paper_title: Probabilistic Map-based Pedestrian Motion Prediction Taking Traffic Participants into Consideration

paper_content:

As pedestrians are one of the most vulnerable traffic participants, their motion prediction is of utmost importance for intelligent transportation systems. Predicting motions of pedestrians is especially hard since they move in less structured environments and have less inertia compared to road vehicles. To account for this uncertainty, we present an approach for probabilistic prediction of pedestrian motion using Markov chains. In contrast to previous work, we not only consider motion models, constraints from a semantic map, and various goals, but also explicitly adapt the prediction based on crash probabilities with other traffic participants. Also, our approach works in any situation; this is typically challenging for pure machine learning techniques that learn behaviors for a particular road section and which might consequently struggle with a different road section. The usefulness of combining the aforementioned aspects in a single approach is demonstrated by an evaluation using recordings of real pedestrians.
---
paper_title: Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning

paper_content:

Finding feasible, collision-free paths for multiagent systems can be challenging, particularly in non-communicating scenarios where each agent's intent (e.g. goal) is unobservable to the others. In particular, finding time efficient paths often requires anticipating interaction with neighboring agents, the process of which can be computationally prohibitive. This work presents a decentralized multiagent collision avoidance algorithm based on a novel application of deep reinforcement learning, which effectively offloads the online computation (for predicting interaction patterns) to an offline learning procedure. Specifically, the proposed approach develops a value network that encodes the estimated time to the goal given an agent's joint configuration (positions and velocities) with its neighbors. Use of the value network not only admits efficient (i.e., real-time implementable) queries for finding a collision-free velocity vector, but also considers the uncertainty in the other agents' motion. Simulation results show more than 26 percent improvement in paths quality (i.e., time to reach the goal) when compared with optimal reciprocal collision avoidance (ORCA), a state-of-the-art collision avoidance strategy.
---
paper_title: Monte Carlo road safety reasoning

paper_content:

This paper presents a framework for reasoning about the future motion of multiple objects in a road scene. Unlike previous approaches, we do not look for known dangerous configurations of objects, but rather we reason about the future paths of all objects in the scene, and assess their danger. Monte Carlo path planning is used to generate a probability distribution for the possible future motion of every car in the scene. This framework can be used to either control the car, or to display warnings for the driver.
---
paper_title: Goal-Directed Pedestrian Prediction

paper_content:

Recent advances in road safety have lead to a constant decline of injured traffic participants in Europe per year. Still, the number of injured pedestrians remains nearly constant. As a countermeasure, active pedestrian safety is the focus of current research, for which accurate pedestrian prediction is a prerequisite. In this scope, we propose a method for dynamics-and environment-based pedestrian prediction. We introduce the pedestrian's destination as a latent variable and thus convert the prediction problem into a planning problem. The planning is executed based on the current dynamics of the pedestrian. The distribution over the destinations is modeled using a Particle Filter. Experimental results show a significant improvement over existing approaches such as Kalman Filters.
---
paper_title: Online trajectory prediction and planning for social robot navigation

paper_content:

This paper addresses the safe and legible navigation of mobile robots in multi-agent encounters. A novel motion model provides the basis to predict, plan and coordinate agent trajectories in intersection scenarios. The approach establishes an implicit, non-overt cooperation between the robot and humans by linking the prediction and planning of agent trajectories within a unified representation in terms of timed elastic bands. The planning process maintains multiple topological alternatives to resolve the encounter in a manner compliant with the implicit rules and objectives of human proxemics. The trajectory is obtained by optimizing the timed elastic band considering multiple conflicting objectives such as fastest path and minimal spatial separation among agents but also global proxemic aspects such as motion coherence within a group. Cooperation is achieved by coupling predicted and planned agent trajectories to eventually reach an implicit agreement of the agents on how to circumnavigate each other. The parameters of the cost functions of the underlying motion model are identified by inverse optimal control from a dataset of 73 recorded encounters with up to five humans and a total of 283 individual trajectories. Playback simulations of recorded encounters and experiments with a robot traversing a group of oncoming humans demonstrate the feasibility of the approach to resolve general proxemic encounters.
---
paper_title: An Algorithmic Perspective on Imitation Learning

paper_content:

As robots and other intelligent agents move from simple environments and problems to more complex, unstructured settings, manually programming their behavior has become increasingly challenging and expensive. Often, it is easier for a teacher to demonstrate a desired behavior rather than attempt to manually engineer it. This process of learning from demonstrations, and the study of algorithms to do so, is called imitation learning. This work provides an introduction to imitation learning. It covers the underlying assumptions, approaches, and how they relate; the rich set of algorithms developed to tackle the problem; and advice on effective tools and implementation. We intend this paper to serve two audiences. First, we want to familiarize machine learning experts with the challenges of imitation learning, particularly those arising in robotics, and the interesting theoretical and practical distinctions between it and more familiar frameworks like statistical supervised learning theory and reinforcement learning. Second, we want to give roboticists and experts in applied artificial intelligence a broader appreciation for the frameworks and tools available for imitation learning. We pay particular attention to the intimate connection between imitation learning approaches and those of structured prediction Daume III et al. [2009]. To structure this discussion, we categorize imitation learning techniques based on the following key criteria which drive algorithmic decisions: ::: ::: 1) The structure of the policy space. Is the learned policy a time-index trajectory (trajectory learning), a mapping from observations to actions (so called behavioral cloning [Bain and Sammut, 1996]), or the result of a complex optimization or planning problem at each execution as is common in inverse optimal control methods [Kalman, 1964, Moylan and Anderson, 1973]. ::: ::: 2) The information available during training and testing. In particular, is the learning algorithm privy to the full state that the teacher possess? Is the learner able to interact with the teacher and gather corrections or more data? Does the learner have a (typically a priori) model of the system with which it interacts? Does the learner have access to the reward (cost) function that the teacher is attempting to optimize? ::: ::: 3) The notion of success. Different algorithmic approaches provide varying guarantees on the resulting learned behavior. These guarantees range from weaker (e.g., measuring disagreement with the agent’s decision) to stronger (e.g., providing guarantees on the performance of the learner with respect to a true cost function, either known or unknown). We organize our work by paying particular attention to distinction (1): dividing imitation learning into directly replicating desired behavior (sometimes called behavioral cloning) and learning the hidden objectives of the desired behavior from demonstrations (called inverse optimal control or inverse reinforcement learning [Russell, 1998]). In the latter case, behavior arises as the result of an optimization problem solved for each new instance that the learner faces. In addition to method analysis, we discuss the design decisions a practitioner must make when selecting an imitation learning approach. Moreover, application examples—such as robots that play table tennis [Kober and Peters, 2009], programs that play the game of Go [Silver et al., 2016], and systems that understand natural language [Wen et al., 2015]— illustrate the properties and motivations behind different forms of imitation learning. We conclude by presenting a set of open questions and point towards possible future research directions for machine learning.
---
paper_title: Planning-based prediction for pedestrians

paper_content:

We present a novel approach for determining robot movements that efficiently accomplish the robot's tasks while not hindering the movements of people within the environment. Our approach models the goal-directed trajectories of pedestrians using maximum entropy inverse optimal control. The advantage of this modeling approach is the generality of its learned cost function to changes in the environment and to entirely different environments. We employ the predictions of this model of pedestrian trajectories in a novel incremental planner and quantitatively show the improvement in hindrance-sensitive robot trajectory planning provided by our approach.
---
paper_title: Transferable Pedestrian Motion Prediction Models at Intersections

paper_content:

One desirable capability of autonomous cars is to accurately predict the pedestrian motion near intersections for safe and efficient trajectory planning. We are interested in developing transfer learning algorithms that can be trained on the pedestrian trajectories collected at one intersection and yet still provide accurate predictions of the trajectories at another, previously unseen intersection. We first discussed the feature selection for transferable pedestrian motion models in general. Following this discussion, we developed one transferable pedestrian motion prediction algorithm based on Inverse Reinforcement Learning (IRL) that infers pedestrian intentions and predicts future trajectories based on observed trajectory. We evaluated our algorithm on a dataset collected at two intersections, trained at one intersection and tested at the other intersection. We used the accuracy of augmented semi-nonnegative sparse coding (ASNSC), trained and tested at the same intersection as a baseline. The result shows that the proposed algorithm improves the baseline accuracy by 40% in the non-transfer task, and 16% in the transfer task.
---
paper_title: Pedestrian Prediction by Planning Using Deep Neural Networks

paper_content:

Accurate traffic participant prediction is the prerequisite for collision avoidance of autonomous vehicles. In this work, we propose to predict pedestrians using goal-directed planning. For this, we infer a mixture density function for possible destinations. We use these destinations as the goal states of a planning stage that performs motion prediction based on common behavior patterns. The patterns are learned by a fully convolutional network operating on maps of the environment. We show that this entire system can be modeled as one monolithic neural network and trained via inverse reinforcement learning. Experimental validation on real world data shows the system's ability to predict both, destinations and trajectories accurately.
---
paper_title: Deep Learning Driven Visual Path Prediction from a Single Image

paper_content:

Capabilities of inference and prediction are the significant components of visual systems. Visual path prediction is an important and challenging task among them, with the goal to infer the future path of a visual object in a static scene. This task is complicated as it needs high-level semantic understandings of both the scenes and underlying motion patterns in video sequences. In practice, cluttered situations have also raised higher demands on the effectiveness and robustness of models. Motivated by these observations, we propose a deep learning framework, which simultaneously performs deep feature learning for visual representation in conjunction with spatiotemporal context modeling. After that, a unified path-planning scheme is proposed to make accurate path prediction based on the analytic results returned by the deep context models. The highly effective visual representation and deep context models ensure that our framework makes a deep semantic understanding of the scenes and motion patterns, consequently improving the performance on visual path prediction task. In experiments, we extensively evaluate the model’s performance by constructing two large benchmark datasets from the adaptation of video tracking datasets. The qualitative and quantitative experimental results show that our approach outperforms the state-of-the-art approaches and owns a better generalization capability.
---
paper_title: Predicting Future Agent Motions for Dynamic Environments

paper_content:

Understanding activities of people in a monitored environment is a topic of active research, motivated by applications requiring context-awareness. Inferring future agent motion is useful not only for improving tracking accuracy, but also for planning in an interactive motion task. Despite rapid advances in the area of activity forecasting, many state-of-the-art methods are still cumbersome for use in realistic robots. This is due to the requirement of having good semantic scene and map labelling, as well as assumptions made regarding possible goals and types of motion. Many emerging applications require robots with modest sensory and computational ability to robustly perform such activity forecasting in high density and dynamic environments. We address this by combining a novel multi-camera tracking method, efficient multi-resolution representations of state and a standard Inverse Reinforcement Learning (IRL) technique, to demonstrate performance that is better than the state-of-the-art in the literature. In this framework, the IRL method uses agent trajectories from a distributed tracker and estimates a reward function within a Markov Decision Process (MDP) model. This reward function can then be used to estimate the agent's motion in future novel task instances. We present empirical experiments using data gathered in our own lab and external corpora (VIRAT), based on which we find that our algorithm is not only efficiently implementable on a resource constrained platform but is also competitive in terms of accuracy with state-of-the-art alternatives (e.g., up to 20% better than the results reported in [1]).
---
paper_title: Activity forecasting

paper_content:

We address the task of inferring the future actions of people from noisy visual input. We denote this task activity forecasting. To achieve accurate activity forecasting, our approach models the effect of the physical environment on the choice of human actions. This is accomplished by the use of state-of-the-art semantic scene understanding combined with ideas from optimal control theory. Our unified model also integrates several other key elements of activity analysis, namely, destination forecasting, sequence smoothing and transfer learning. As proof-of-concept, we focus on the domain of trajectory-based activity analysis from visual input. Experimental results demonstrate that our model accurately predicts distributions over future actions of individuals. We show how the same techniques can improve the results of tracking algorithms by leveraging information about likely goals and trajectories.
---
paper_title: Imitating driver behavior with generative adversarial networks

paper_content:

The ability to accurately predict and simulate human driving behavior is critical for the development of intelligent transportation systems. Traditional modeling methods have employed simple parametric models and behavioral cloning. This paper adopts a method for overcoming the problem of cascading errors inherent in prior approaches, resulting in realistic behavior that is robust to trajectory perturbations. We extend Generative Adversarial Imitation Learning to the training of recurrent policies, and we demonstrate that our model rivals rule-based controllers and maximum likelihood models in realistic highway simulations. Our model both reproduces emergent behavior of human drivers, such as lane change rate, while maintaining realistic control over long time horizons.
---
paper_title: InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations

paper_content:

The goal of imitation learning is to mimic expert behavior without access to an explicit reward signal. Expert demonstrations provided by humans, however, often show significant variability due to latent factors that are typically not explicitly modeled. In this paper, we propose a new algorithm that can infer the latent structure of expert demonstrations in an unsupervised way. Our method, built on top of Generative Adversarial Imitation Learning, can not only imitate complex behaviors, but also learn interpretable and meaningful representations of complex behavioral data, including visual demonstrations. In the driving domain, we show that a model learned from human demonstrations is able to both accurately reproduce a variety of behaviors and accurately anticipate human actions using raw visual inputs. Compared with various baselines, our method can better capture the latent structure underlying expert demonstrations, often recovering semantically meaningful factors of variation in the data.
---
paper_title: Generative Adversarial Imitation Learning

paper_content:

Consider learning a policy from example expert behavior, without interaction with the expert or access to reinforcement signal. One approach is to recover the expert's cost function with inverse reinforcement learning, then extract a policy from that cost function with reinforcement learning. This approach is indirect and can be slow. We propose a new general framework for directly extracting a policy from data, as if it were obtained by reinforcement learning following inverse reinforcement learning. We show that a certain instantiation of our framework draws an analogy between imitation learning and generative adversarial networks, from which we derive a model-free imitation learning algorithm that obtains significant performance gains over existing model-free methods in imitating complex behaviors in large, high-dimensional environments.
---
paper_title: Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks

paper_content:

Understanding human motion behavior is critical for autonomous moving platforms (like self-driving cars and social robots) if they are to navigate human-centric environments. This is challenging because human motion is inherently multimodal: given a history of human motion paths, there are many socially plausible ways that people could move in the future. We tackle this problem by combining tools from sequence prediction and generative adversarial networks: a recurrent sequence-to-sequence model observes motion histories and predicts future behavior, using a novel pooling mechanism to aggregate information across people. We predict socially plausible futures by training adversarially against a recurrent discriminator, and encourage diverse predictions with a novel variety loss. Through experiments on several datasets we demonstrate that our approach outperforms prior work in terms of accuracy, variety, collision avoidance, and computational complexity.
---
paper_title: Generative Adversarial Nets

paper_content:

We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to ½ everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.
---
paper_title: Planning-based prediction for pedestrians

paper_content:

We present a novel approach for determining robot movements that efficiently accomplish the robot's tasks while not hindering the movements of people within the environment. Our approach models the goal-directed trajectories of pedestrians using maximum entropy inverse optimal control. The advantage of this modeling approach is the generality of its learned cost function to changes in the environment and to entirely different environments. We employ the predictions of this model of pedestrian trajectories in a novel incremental planner and quantitatively show the improvement in hindrance-sensitive robot trajectory planning provided by our approach.
---
paper_title: Forecasting Interactive Dynamics of Pedestrians with Fictitious Play

paper_content:

We develop predictive models of pedestrian dynamics by encoding the coupled nature of multi-pedestrian interaction using game theory, and deep learning-based visual analysis to estimate person-specific behavior parameters. Building predictive models for multi-pedestrian interactions however, is very challenging due to two reasons: (1) the dynamics of interaction are complex interdependent processes, where the predicted behavior of one pedestrian can affect the actions taken by others and (2) dynamics are variable depending on an individuals physical characteristics (e.g., an older person may walk slowly while the younger person may walk faster). To address these challenges, we (1) utilize concepts from game theory to model the interdependent decision making process of multiple pedestrians and (2) use visual classifiers to learn a mapping from pedestrian appearance to behavior parameters. We evaluate our proposed model on several public multiple pedestrian interaction video datasets. Results show that our strategic planning model explains human interactions 25% better when compared to state-of-the-art methods.
---
paper_title: Learning to predict trajectories of cooperatively navigating agents

paper_content:

The problem of modeling the navigation behavior of multiple interacting agents arises in different areas including robotics, computer graphics, and behavioral science. In this paper, we present an approach to learn the composite navigation behavior of interacting agents from demonstrations. The decision process that ultimately leads to the observed continuous trajectories of the agents often also comprises discrete decisions, which partition the space of composite trajectories into homotopy classes. Therefore, our method uses a mixture probability distribution that consists of a discrete distribution over the homotopy classes and continuous distributions over the composite trajectories for each homotopy class. Our approach learns the model parameters of this distribution that match, in expectation, the observed behavior in terms of user-defined features. To compute the feature expectations over the highdimensional continuous distributions, we use Hamiltonian Markov chain Monte Carlo sampling. We exploit that the distributions are highly structured due to physical constraints and guide the sampling process to regions of high probability. We apply our approach to learning the behavior of pedestrians and demonstrate that it outperforms state-of-the-art methods.
---
paper_title: Learning to navigate through crowded environments

paper_content:

The goal of this research is to enable mobile robots to navigate through crowded environments such as indoor shopping malls, airports, or downtown side walks. The key research question addressed in this paper is how to learn planners that generate human-like motion behavior. Our approach uses inverse reinforcement learning (IRL) to learn human-like navigation behavior based on example paths. Since robots have only limited sensing, we extend existing IRL methods to the case of partially observable environments. We demonstrate the capabilities of our approach using a realistic crowd flow simulator in which we modeled multiple scenarios in crowded environments. We show that our planner learned to guide the robot along the flow of people when the environment is crowded, and along the shortest path if no people are around.
---
paper_title: DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents

paper_content:

We introduce a Deep Stochastic IOC RNN Encoder-decoder framework, DESIRE, for the task of future predictions of multiple interacting agents in dynamic scenes. DESIRE effectively predicts future locations of objects in multiple scenes by 1) accounting for the multi-modal nature of the future prediction (i.e., given the same context, future may vary), 2) foreseeing the potential future outcomes and make a strategic prediction based on that, and 3) reasoning not only from the past motion history, but also from the scene context as well as the interactions among the agents. DESIRE achieves these in a single end-to-end trainable neural network model, while being computationally efficient. The model first obtains a diverse set of hypothetical future prediction samples employing a conditional variational auto-encoder, which are ranked and refined by the following RNN scoring-regression module. Samples are scored by accounting for accumulated future rewards, which enables better long-term strategic decisions similar to IOC frameworks. An RNN scene context fusion module jointly captures past motion histories, the semantic scene context and interactions among multiple agents. A feedback mechanism iterates over the ranking and refinement to further boost the prediction accuracy. We evaluate our model on two publicly available datasets: KITTI and Stanford Drone Dataset. Our experiments show that the proposed model significantly improves the prediction accuracy compared to other baseline methods.
---
paper_title: Patch to the Future: Unsupervised Visual Prediction

paper_content:

In this paper we present a conceptually simple but surprisingly powerful method for visual prediction which combines the effectiveness of mid-level visual elements with temporal modeling. Our framework can be learned in a completely unsupervised manner from a large collection of videos. However, more importantly, because our approach models the prediction framework on these mid-level elements, we can not only predict the possible motion in the scene but also predict visual appearances--how are appearances going to change with time. This yields a visual "hallucination" of probable events on top of the scene. We show that our method is able to accurately predict and visualize simple future events, we also show that our approach is comparable to supervised methods for event prediction.
---
paper_title: Predicting actions to act predictably: Cooperative partial motion planning with maximum entropy models

paper_content:

This paper reports on a data-driven motion planning approach for interaction-aware, socially-compliant robot navigation among human agents. Autonomous mobile robots navigating in workspaces shared with human agents require motion planning techniques providing seamless integration and smooth navigation in such. Smooth integration in mixed scenarios calls for two abilities of the robot: predicting actions of others and acting predictably for them. The former requirement requests trainable models of agent behaviors in order to accurately forecast their actions in the future, taking into account their reaction on the robot's decisions. A human-like navigation style of the robot facilitates other agents—most likely not aware of the underlying planning technique applied—to predict the robot motion vice versa, resulting in smoother joint navigation. The approach presented in this paper is based on a feature-based maximum entropy model and is able to guide a robot in an unstructured, real-world environment. The model is trained to predict joint behavior of heterogeneous groups of agents from onboard data of a mobile platform. We evaluate the benefit of interaction-aware motion planning in a realistic public setting with a total distance traveled of over 4 km. Interestingly the motion models learned from human-human interaction did not hold for robot-human interaction, due to the high attention and interest of pedestrians in testing basic braking functionality of the robot.
---
paper_title: Predicting wide receiver trajectories in American football

paper_content:

Predicting the trajectory of a wide receiver in the game of American football requires prior knowledge about the game (e.g., route trees, defensive formations) and an accurate model of how the environment will change over time (e.g., opponent reaction strategies, motion attributes of players). Our aim is to build a computational model of the wide receiver, which takes into account prior knowledge about the game and short-term predictive models of how the environment will change over time. While prior knowledge of the game is readily accessible, it is quite challenging to build predictive models of how the environment will change over time. We propose several models for predicting short-term motions of opponent players to generate dynamic input features for our wide receiver forecasting model. In particular, we model the wide receiver with a Markov Decision Process (MDP), where the reward function is a linear combination of static features (prior knowledge about the game) and dynamic features (short-term prediction of opponent players). Since the dynamic features change over time, we make recursive calls to an inference procedure over the MDP while updating the dynamic features. We validate our technique on a video dataset of American football plays. Our results show that more informed models that accurately predict the motions of the defensive players are better at forecasting wide receiver plays.
---
paper_title: Behavior estimation for a complete framework for human motion prediction in crowded environments

paper_content:

In the present work, we propose and validate a complete probabilistic framework for human motion prediction in urban or social environments. Additionally, we formulate a powerful and useful tool: the human motion behavior estimator. Three different basic behaviors have been detected: Aware, Balanced and Unaware. Our approach is based on the Social Force Model (SFM) and the intentionality prediction BHMIP. The main contribution of the present work is to make use of the behavior estimator for formulating a reliable prediction framework of human trajectories under the influence of dynamic crowds, robots, and in general any moving obstacle. Accordingly, we have demonstrated the great performance of our long-term prediction algorithm, in real scenarios, comparing to other prediction methods.
---
paper_title: Unfreezing the robot: Navigation in dense, interacting crowds

paper_content:

In this paper, we study the safe navigation of a mobile robot through crowds of dynamic agents with uncertain trajectories. Existing algorithms suffer from the “freezing robot” problem: once the environment surpasses a certain level of complexity, the planner decides that all forward paths are unsafe, and the robot freezes in place (or performs unnecessary maneuvers) to avoid collisions. Since a feasible path typically exists, this behavior is suboptimal. Existing approaches have focused on reducing the predictive uncertainty for individual agents by employing more informed models or heuristically limiting the predictive covariance to prevent this overcautious behavior. In this work, we demonstrate that both the individual prediction and the predictive uncertainty have little to do with the frozen robot problem. Our key insight is that dynamic agents solve the frozen robot problem by engaging in “joint collision avoidance”: They cooperatively make room to create feasible trajectories. We develop IGP, a nonparametric statistical model based on dependent output Gaussian processes that can estimate crowd interaction from data. Our model naturally captures the non-Markov nature of agent trajectories, as well as their goal-driven navigation. We then show how planning in this model can be efficiently implemented using particle based inference. Lastly, we evaluate our model on a dataset of pedestrians entering and leaving a building, first comparing the model with actual pedestrians, and find that the algorithm either outperforms human pedestrians or performs very similarly to the pedestrians. We also present an experiment where a covariance reduction method results in highly overcautious behavior, while our model performs desirably.
---
paper_title: Knowledge Transfer for Scene-specific Motion Prediction

paper_content:

When given a single frame of the video, humans can not only interpret the content of the scene, but also they are able to forecast the near future. This ability is mostly driven by their rich prior knowledge about the visual world, both in terms of (i) the dynamics of moving agents, as well as (ii) the semantic of the scene. In this work we exploit the interplay between these two key elements to predict scene-specific motion patterns. First, we extract patch descriptors encoding the probability of moving to the adjacent patches, and the probability of being in that particular patch or changing behavior. Then, we introduce a Dynamic Bayesian Network which exploits this scene specific knowledge for trajectory prediction. Experimental results demonstrate that our method is able to accurately predict trajectories and transfer predictions to a novel scene characterized by similar elements.
---
paper_title: Context-Based Path Prediction for Targets with Switching Dynamics

paper_content:

Anticipating future situations from streaming sensor data is a key perception challenge for mobile robotics and automated vehicles. We address the problem of predicting the path of objects with multiple dynamic modes. The dynamics of such targets can be described by a Switching Linear Dynamical System (SLDS). However, predictions from this probabilistic model cannot anticipate when a change in dynamic mode will occur. We propose to extract various types of cues with computer vision to provide context on the target’s behavior, and incorporate these in a Dynamic Bayesian Network (DBN). The DBN extends the SLDS by conditioning the mode transition probabilities on additional context states. We describe efficient online inference in this DBN for probabilistic path prediction, accounting for uncertainty in both measurements and target behavior. Our approach is illustrated on two scenarios in the Intelligent Vehicles domain concerning pedestrians and cyclists, so-called Vulnerable Road Users (VRUs). Here, context cues include the static environment of the VRU, its dynamic environment, and its observed actions. Experiments using stereo vision data from a moving vehicle demonstrate that the proposed approach results in more accurate path prediction than SLDS at the relevant short time horizon (1 s). It slightly outperforms a computationally more demanding state-of-the-art method.
---
paper_title: A Controlled Interactive Multiple Model Filter for Combined Pedestrian Intention Recognition and Path Prediction

paper_content:

We present a novel approach combining pedestrian intention recognition and path prediction for advanced video-based driver assistance systems. The core algorithm uses an Interacting Multiple Model Filter in combination with a Latent-dynamic Conditional Random Field model. The model integrates pedestrian dynamics and situational awareness using observations from a stereo-video system for pedestrian detection and human head pose estimation. Evaluation of our method is performed on a public available dataset addressing scenarios of lateral approaching pedestrians that might cross the road, turn into the road or stop at the curbside. During experiments, we demonstrate that the proposed approach leads to better path prediction performance in terms of a smaller lateral position error compared to state-of-the-art pedestrian intention recognition and path prediction approaches. The computational costs of our approach is comparatively low and therefore can be ported easily onto a real-time system.
---
paper_title: Planning-based prediction for pedestrians

paper_content:

We present a novel approach for determining robot movements that efficiently accomplish the robot's tasks while not hindering the movements of people within the environment. Our approach models the goal-directed trajectories of pedestrians using maximum entropy inverse optimal control. The advantage of this modeling approach is the generality of its learned cost function to changes in the environment and to entirely different environments. We employ the predictions of this model of pedestrian trajectories in a novel incremental planner and quantitatively show the improvement in hindrance-sensitive robot trajectory planning provided by our approach.
---
paper_title: Enabling Flow Awareness for Mobile Robots in Partially Observable Environments

paper_content:

Understanding the environment is a key requirement for any autonomous robot operation. There is extensive research on mapping geometric structure and perceiving objects. However, the environment is ...
---
paper_title: Human Motion Prediction Under Social Grouping Constraints

paper_content:

Accurate long-term prediction of human motion in populated spaces is an important but difficult task for mobile robots and intelligent vehicles. What makes this task challenging is that human motion is influenced by a large variety of factors including the person's intention, the presence, attributes, actions, social relations and social norms of other surrounding agents, and the geometry and semantics of the environment. In this paper, we consider the problem of computing human motion predictions that account for such factors. We formulate the task as an MDP planning problem with stochastic policies and propose a weighted random walk algorithm in which each agent is locally influenced by social forces from other nearby agents. The novelty of this paper is that we incorporate social grouping information into the prediction process reflecting the soft formation constraints that groups typically impose to their members' motion. We show that our method makes more accurate predictions than three state-of-the-art methods in terms of probabilistic and geometrical performance metrics.
---
paper_title: Long-term vehicle motion prediction

paper_content:

Future driver assistance systems will have to cope with complex traffic situations, especially in the road crossing scenario. To detect potentially hazardous situations as early as possible, it is therefore desirable to know the position and motion of the ego-vehicle and vehicles around it for several seconds in advance. For this purpose, we propose in this study a long-term prediction approach based on a combined trajectory classification and particle filter framework. As a measure for the similarity between trajectories, we introduce the quaternion-based rotationally invariant longest common subsequence (QRLCS) metric. The trajectories are classified by a radial basis function (RBF) classifier with an architecture that is able to process trajectories of arbitrary non-uniform length. The particle filter framework simultaneously tracks and assesses a large number of motion hypotheses (∼102), where the class-specific probabilities estimated by the RBF classifier are used as a-priori probabilities for the hypotheses of the particle filter. The hypotheses are clustered with a mean-shift technique and are assigned a likelihood value. Motion prediction is performed based on the cluster centre with the highest likelihood. While traditional motion prediction based on curve radius and acceleration is inaccurate especially during turning manoeuvres, we show that our approach achieves a reasonable motion prediction even for long prediction intervals of 3 s for these complex motion patterns.
---
paper_title: Intent-aware long-term prediction of pedestrian motion

paper_content:

We present a method to predict long-term motion of pedestrians, modeling their behavior as jump-Markov processes with their goal a hidden variable. Assuming approximately rational behavior, and incorporating environmental constraints and biases, including time-varying ones imposed by traffic lights, we model intent as a policy in a Markov decision process framework. We infer pedestrian state using a Rao-Blackwellized filter, and intent by planning according to a stochastic policy, reflecting individual preferences in aiming at the same goal.
---
paper_title: Learning intentions for improved human motion prediction

paper_content:

For many tasks, robots need to operate in human populated environments. Human motion prediction therefore is gaining importance. The concept of social forces defines virtual repelling and attracting forces from and to obstacles and points of interest. These social forces can then be used to model typical human movements given an environment and a person's intention. This work shows how such models can exploit typical motion patterns summarized by growing hidden Markov models (GHMMs) that can be learned from data online and without human intervention. An extensive series of experiments shows that exploiting the intended position estimated using a GHMM within a social forces based motion model yields a significant performance gain in comparison with the standard constant velocity-based models.
---
paper_title: Forecasting Interactive Dynamics of Pedestrians with Fictitious Play

paper_content:

We develop predictive models of pedestrian dynamics by encoding the coupled nature of multi-pedestrian interaction using game theory, and deep learning-based visual analysis to estimate person-specific behavior parameters. Building predictive models for multi-pedestrian interactions however, is very challenging due to two reasons: (1) the dynamics of interaction are complex interdependent processes, where the predicted behavior of one pedestrian can affect the actions taken by others and (2) dynamics are variable depending on an individuals physical characteristics (e.g., an older person may walk slowly while the younger person may walk faster). To address these challenges, we (1) utilize concepts from game theory to model the interdependent decision making process of multiple pedestrians and (2) use visual classifiers to learn a mapping from pedestrian appearance to behavior parameters. We evaluate our proposed model on several public multiple pedestrian interaction video datasets. Results show that our strategic planning model explains human interactions 25% better when compared to state-of-the-art methods.
---
paper_title: Long-term Path Prediction in Urban Scenarios using Circular Distributions

paper_content:

Abstract Human ability to foresee the near future plays a key role in everyone's life to prevent potentially dangerous situations. To be able to make predictions is crucial when people have to interact with the surrounding environment. Modeling such capability can lead to the design of automated warning systems and provide moving robots with an intelligent way of interaction with changing situation. In this work we focus on a typical urban human-scene where we aim at predicting an agent's behavior using a stochastic model. In this approach, we fuse the various factors that would contribute to a human motion in different contexts. Our method uses previously observed trajectories to build point-wise circular distributions that after combination, provide a statistical smooth prediction towards the most likely areas. More specifically, a ray-launching procedure, based on a semantic segmentation, gives a coarse scene representation for collision avoidance; a nearly-constant velocity dynamic model smooths the acceleration progression and knowledge of the agent's destination may further steer the path prediction. Experimental results in structured scenes, validate the effectiveness of the method in predicting paths in comparison to actual trajectories.
---
paper_title: A Game-Theoretic Approach to Replanning-Aware Interactive Scene Prediction and Planning

paper_content:

This paper presents a novel cooperative-driving prediction and planning framework for dynamic environments based on the methods of game theory. The proposed algorithm can be used for highly automated driving on highways or as a sophisticated prediction module for advanced driver-assistance systems with no need for intervehicle communication. The main contribution of this paper is a model-based interaction-aware motion prediction of all vehicles in a scene. In contrast to other state-of-the-art approaches, the system also models the replanning capabilities of all drivers. With that, the driving strategy is able to capture complex interactions between vehicles, thus planning maneuver sequences over longer time horizons. It also enables an accurate prediction of traffic for the next immediate time step. The prediction model is supported by an interpretation of what other drivers intend to do, how they interact with traffic, and the ongoing observation. As part of the prediction loop, the proposed planning strategy incorporates the expected reactions of all traffic participants, offering cooperative and robust driving decisions. By means of experimental results under simulated highway scenarios, the validity of the proposed concept and its real-time capability is demonstrated.
---
paper_title: Recognition of situation classes at road intersections

paper_content:

The recognition and prediction of situations is an indispensable skill of future driver assistance systems. This study focuses on the recognition of situations involving two vehicles at intersections. For each vehicle, a set of possible future motion trajectories is estimated and rated based on a motion database for a time interval of 2–4 seconds ahead. Realistic situations are generated by a pairwise combination of these individual motion trajectories and classified according to nine categories with a polynomial classifier. In the proposed framework, situations are penalised for which the time to collision significantly exceeds the typical human reaction time. The correspondingly favoured situations are combined by a probabilistic framework, resulting in a more reliable situation recognition and collision detection than obtained based on independent motion hypotheses. The proposed method is evaluated on a real-world differential GPS data set acquired during a test drive of 10 km, including three road intersections. Our method is typically able to recognise the situation correctly about 1–2 seconds before the distance to the intersection centre becomes minimal.
---
paper_title: Context-Aware Pedestrian Motion Prediction In Urban Intersections

paper_content:

This paper presents a novel context-based approach for pedestrian motion prediction in crowded, urban intersections, with the additional flexibility of prediction in similar, but new, environments. Previously, Chen et. al. combined Markovian-based and clustering-based approaches to learn motion primitives in a grid-based world and subsequently predict pedestrian trajectories by modeling the transition between learned primitives as a Gaussian Process (GP). This work extends that prior approach by incorporating semantic features from the environment (relative distance to curbside and status of pedestrian traffic lights) in the GP formulation for more accurate predictions of pedestrian trajectories over the same timescale. We evaluate the new approach on real-world data collected using one of the vehicles in the MIT Mobility On Demand fleet. The results show 12.5% improvement in prediction accuracy and a 2.65 times reduction in Area Under the Curve (AUC), which is used as a metric to quantify the span of predicted set of trajectories, such that a lower AUC corresponds to a higher level of confidence in the future direction of pedestrian motion.
---
paper_title: GLMP- realtime pedestrian path prediction using global and local movement patterns

paper_content:

We present a novel real-time algorithm to predict the path of pedestrians in cluttered environments. Our approach makes no assumption about pedestrian motion or crowd density, and is useful for short-term as well as long-term prediction. We interactively learn the characteristics of pedestrian motion and movement patterns from 2D trajectories using Bayesian inference. These include local movement patterns corresponding to the current and preferred velocities and global characteristics such as entry points and movement features. Our approach involves no precomputation and we demonstrate the real-time performance of our prediction algorithm on sparse and noisy trajectory data extracted from dense indoor and outdoor crowd videos. The combination of local and global movement patterns can improve the accuracy of long-term prediction by 12–18% over prior methods in high-density videos.
---
paper_title: Probabilistic Map-based Pedestrian Motion Prediction Taking Traffic Participants into Consideration

paper_content:

As pedestrians are one of the most vulnerable traffic participants, their motion prediction is of utmost importance for intelligent transportation systems. Predicting motions of pedestrians is especially hard since they move in less structured environments and have less inertia compared to road vehicles. To account for this uncertainty, we present an approach for probabilistic prediction of pedestrian motion using Markov chains. In contrast to previous work, we not only consider motion models, constraints from a semantic map, and various goals, but also explicitly adapt the prediction based on crash probabilities with other traffic participants. Also, our approach works in any situation; this is typically challenging for pure machine learning techniques that learn behaviors for a particular road section and which might consequently struggle with a different road section. The usefulness of combining the aforementioned aspects in a single approach is demonstrated by an evaluation using recordings of real pedestrians.
---
paper_title: Human Motion Behaviour Aware Planner (HMBAP) for path planning in dynamic human environments

paper_content:

For a robot navigating in a human inhabited dynamic environment, the knowledge of how the robot's movement can influence the trajectory of people around it can be very valuable. In this work we present a Human Motion Behaviour Aware Planner (HMBAP) which incorporates a Human Motion Behaviour Model (HMBM) in its planning stage to take advantage of this. HMBM is an obstacle avoidance model for people based on social forces which gives the robot an understanding of how people would react to its planned path. This information is useful for the robot to avoid imminent collisions with people in constricted spaces. The resulting robot behaviour is similar to how a human would move (in terms of avoidance behaviour) in a similar situation. We believe that this is a desirable feature for a robot navigating in a human inhabited environment. Our method shows good human-like navigation behaviour in situations where past methods fail to find a solution.
---
paper_title: Human-robot co-navigation using anticipatory indicators of human walking motion

paper_content:

Mobile, interactive robots that operate in human-centric environments need the capability to safely and efficiently navigate around humans. This requires the ability to sense and predict human motion trajectories and to plan around them. In this paper, we present a study that supports the existence of statistically significant biomechanical turn indicators of human walking motions. Further, we demonstrate the effectiveness of these turn indicators as features in the prediction of human motion trajectories. Human motion capture data is collected with predefined goals to train and test a prediction algorithm. Use of anticipatory features results in improved performance of the prediction algorithm. Lastly, we demonstrate the closed-loop performance of the prediction algorithm using an existing algorithm for motion planning within dynamic environments. The anticipatory indicators of human walking motion can be used with different prediction and/or planning algorithms for robotics; the chosen planning and prediction algorithm demonstrates one such implementation for human-robot co-navigation.
---
paper_title: Pedestrian Path, Pose, and Intention Prediction Through Gaussian Process Dynamical Models and Pedestrian Activity Recognition

paper_content:

According to several reports published by worldwide organizations, thousands of pedestrians die in road accidents every year. Due to this fact, vehicular technologies have been evolving with the intent of reducing these fatalities. This evolution has not finished yet, since, for instance, the predictions of pedestrian paths could improve the current automatic emergency braking systems. For this reason, this paper proposes a method to predict future pedestrian paths, poses, and intentions up to 1 s in advance. This method is based on balanced Gaussian process dynamical models (B-GPDMs), which reduce the 3-D time-related information extracted from key points or joints placed along pedestrian bodies into low-dimensional spaces. The B-GPDM is also capable of inferring future latent positions and reconstruct their associated observations. However, learning a generic model for all kinds of pedestrian activities normally provides less accurate predictions. For this reason, the proposed method obtains multiple models of four types of activity, i.e., walking, stopping, starting, and standing, and selects the most similar model to estimate future pedestrian states. This method detects starting activities 125 ms after the gait initiation with an accuracy of 80% and recognizes stopping intentions 58.33 ms before the event with an accuracy of 70%. Concerning the path prediction, the mean error for stopping activities at a time-to-event (TTE) of 1 s is 238.01 ± 206.93 mm and, for starting actions, the mean error at a TTE of 0 s is 331.93 ± 254.73 mm.
---
paper_title: Learning Motion Patterns of People for Compliant Robot Motion

paper_content:

Whenever people move through their environments they do not move randomly. Instead, they usually follow specific trajectories or motion patterns corresponding to their intentions. Knowledge about such patterns enables a mobile robot to robustly keep track of persons in its environment and to improve its behavior. In this paper we propose a technique for learning collections of trajectories that characterize typical motion patterns of persons. Data recorded with laser-range finders are clustered using the expectation maximization algorithm. Based on the result of the clustering process, we derive a hidden Markov model that is applied to estimate the current and future positions of persons based on sensory input. We also describe how to incorporate the probabilistic belief about the potential trajectories of persons into the path planning process of a mobile robot. We present several experiments carried out in different environments with a mobile robot equipped with a laser-range scanner and a camera system...
---
paper_title: Monte Carlo road safety reasoning

paper_content:

This paper presents a framework for reasoning about the future motion of multiple objects in a road scene. Unlike previous approaches, we do not look for known dangerous configurations of objects, but rather we reason about the future paths of all objects in the scene, and assess their danger. Monte Carlo path planning is used to generate a probability distribution for the possible future motion of every car in the scene. This framework can be used to either control the car, or to display warnings for the driver.
---
paper_title: Pedestrian path prediction using body language traits

paper_content:

Driver Assistance Systems have achieved a high level of maturity in the latest years. As an example of that, sophisticated pedestrian protection systems are already available in a number of commercial vehicles from several OEMs. However, accurate pedestrian path prediction is needed in order to go a step further in terms of safety and reliability, since it can make the difference between effective and non-effective intervention. In this paper, we consider the three-dimensional pedestrian body language in order to perform path prediction in a probabilistic framework. For this purpose, the different body parts and joints are detected using stereo vision. We propose the use of GPDM (Gaussian Process Dynamical Models) for reducing the high dimensionality of the input feature vector (composed by joints and displacement vectors) in the 3D pose space and for learning the pedestrian dynamics in a latent space. Experimental results show that accurate path prediction can be achieved at a time horizon of ≈ 0.8 s.
---
paper_title: Driver and pedestrian awareness-based collision risk analysis

paper_content:

We present a novel approach for vehicle-pedestrian collision risk analysis that incorporates mutual situational awareness, a degree of potential motion coupling and the spatial layout of the environment. The approach uses a Dynamic Bayesian Network (DBN) for modeling the individual object paths; collision risk is subsequently computed by an intersection operation.
---
paper_title: Activity forecasting

paper_content:

We address the task of inferring the future actions of people from noisy visual input. We denote this task activity forecasting. To achieve accurate activity forecasting, our approach models the effect of the physical environment on the choice of human actions. This is accomplished by the use of state-of-the-art semantic scene understanding combined with ideas from optimal control theory. Our unified model also integrates several other key elements of activity analysis, namely, destination forecasting, sequence smoothing and transfer learning. As proof-of-concept, we focus on the domain of trajectory-based activity analysis from visual input. Experimental results demonstrate that our model accurately predicts distributions over future actions of individuals. We show how the same techniques can improve the results of tracking algorithms by leveraging information about likely goals and trajectories.
---
paper_title: MX-LSTM: Mixing Tracklets and Vislets to Jointly Forecast Trajectories and Head Poses

paper_content:

Recent approaches on trajectory forecasting use tracklets to predict the future positions of pedestrians exploiting Long Short Term Memory (LSTM) architectures. This paper shows that adding vislets, that is, short sequences of head pose estimations, allows to increase significantly the trajectory forecasting performance. We then propose to use vislets in a novel framework called MX-LSTM, capturing the interplay between tracklets and vislets thanks to a joint unconstrained optimization of full covariance matrices during the LSTM backpropagation. At the same time, MX-LSTM predicts the future head poses, increasing the standard capabilities of the long-term trajectory forecasting approaches. With standard head pose estimators and an attentional-based social pooling, MX-LSTM scores the new trajectory forecasting state-of-the-art in all the considered datasets (Zara01, Zara02, UCY, and TownCentre) with a dramatic margin when the pedestrians slow down, a case where most of the forecasting approaches struggle to provide an accurate solution.
---
paper_title: You'll never walk alone: Modeling social behavior for multi-target tracking

paper_content:

Object tracking typically relies on a dynamic model to predict the object's location from its past trajectory. In crowded scenarios a strong dynamic model is particularly important, because more accurate predictions allow for smaller search regions, which greatly simplifies data association. Traditional dynamic models predict the location for each target solely based on its own history, without taking into account the remaining scene objects. Collisions are resolved only when they happen. Such an approach ignores important aspects of human behavior: people are driven by their future destination, take into account their environment, anticipate collisions, and adjust their trajectories at an early stage in order to avoid them. In this work, we introduce a model of dynamic social behavior, inspired by models developed for crowd simulation. The model is trained with videos recorded from birds-eye view at busy locations, and applied as a motion model for multi-people tracking from a vehicle-mounted camera. Experiments on real sequences show that accounting for social interactions and scene knowledge improves tracking performance, especially during occlusions.
---
paper_title: Behavior estimation for a complete framework for human motion prediction in crowded environments

paper_content:

In the present work, we propose and validate a complete probabilistic framework for human motion prediction in urban or social environments. Additionally, we formulate a powerful and useful tool: the human motion behavior estimator. Three different basic behaviors have been detected: Aware, Balanced and Unaware. Our approach is based on the Social Force Model (SFM) and the intentionality prediction BHMIP. The main contribution of the present work is to make use of the behavior estimator for formulating a reliable prediction framework of human trajectories under the influence of dynamic crowds, robots, and in general any moving obstacle. Accordingly, we have demonstrated the great performance of our long-term prediction algorithm, in real scenarios, comparing to other prediction methods.
---
paper_title: Prediction of moving objects in dynamic environments using Kalman filters

paper_content:

In this work, we describe a framework for predicting future positions and orientation of moving obstacles in a time-varying environment using Kalman filtering techniques. No constraint are placed on the obstacles motion. The proposed algorithm can be used in a variety of applications, one of which is robot motion planning in time varying environments. The advantage of using this model compared to reported ones in the literature is it's ability to start the prediction process from the first time step without the need to wait for few time steps before starting the prediction process. Early experimental results are very encouraging.
---
paper_title: A Predictive Collision Avoidance Model for Pedestrian Simulation

paper_content:

We present a new local method for collision avoidance that is based on collision prediction. In our model, each pedestrian predicts possible future collisions with other pedestrians and then makes an efficient move to avoid them. Experiments show that the new approach leads to considerably shorter and less curved paths, ensuring smooth avoidance behaviour and visually compelling simulations. The method reproduces emergent behaviour like lane formation that have been observed in real crowds. The technique is easy to implement and is fast, allowing the simulation in real time of crowds of thousands of pedestrians.
---
paper_title: Sequence-to-Sequence Prediction of Vehicle Trajectory via LSTM Encoder-Decoder Architecture

paper_content:

In this paper, we propose a deep learning based vehicle trajectory prediction technique which can generate the future trajectory sequence of surrounding vehicles in real time. We employ the encoder-decoder architecture which analyzes the pattern underlying in the past trajectory using the long short-term memory (LSTM) based encoder and generates the future trajectory sequence using the LSTM based decoder. This structure produces the $K$ most likely trajectory candidates over occupancy grid map by employing the beam search technique which keeps the $K$ locally best candidates from the decoder output. The experiments conducted on highway traffic scenarios show that the prediction accuracy of the proposed method is significantly higher than the conventional trajectory prediction techniques.
---
paper_title: Intent Prediction of Pedestrians via Motion Trajectories Using Stacked Recurrent Neural Networks

paper_content:

The problem of intent understanding between highly and fully automated vehicles and vulnerable road users (VRUs) such as pedestrians in urban traffic environment has got some momentum over the past few years. Previous work has been tackling the problem using two common approaches, namely dynamical motion modeling and motion planning. In this paper, a novel radical end-to-end data-driven approach is proposed for long-term intent prediction of VRUs in urban traffic environment based solely on their motion trajectories. In the proposed approach, we utilized the widely adopted architecture of recurrent neural networks with Long-Short Term Memory (LSTM) modules to form a deep stacked LSTM network. Three common approaches used in the literature were compared against our proposed approach over two different real-world datasets involving pedestrians collected from vehicle-based stereo cameras. The results over the testing datasets showed that the proposed approach achieved higher accuracies over most of the scenarios of the testing datasets with a small mean lateral position error of 0.48 m. Moreover, the proposed approach showed also a significant generalization capability over totally unobserved testing scenes during the training phase with only 0.58 m in mean lateral position error.
---
paper_title: Trajectory prediction of cyclists using a physical model and an artificial neural network

paper_content:

This article presents two methods for predicting the trajectories of cyclists at an intersection and compares them to a Kalman Filter (KF) approach. The first method uses a physical model of cyclists to predict their future position. The second method is based on a polynomial least-squares approximation in combination with a multilayer perceptron artificial neural network and is able to predict the future position of cyclists independent of their motion type such as “Starting”, “Stopping”, “Waiting” or “Passing”. To evaluate the performance of the methods, 566 tracks (394 for training, 172 for testing) of uninstructed cyclists were recorded at a public intersection using a wide angle stereo camera system and laser scanners. Using the tracks as input data, the future trajectory was predicted for a time horizon of 2.5 s. For starting motions the prediction using the physical model leads to 27% more accurate positions than the KF approach for a forecast horizon of 2.5 s. The neural network shows a 34% more accurate result for starting and stopping motions and a similar result for waiting and passing motions.
---
paper_title: Conditional transition maps: Learning motion patterns in dynamic environments

paper_content:

In this paper we introduce a method for learning motion patterns in dynamic environments. Representations of dynamic environments have recently received an increasing amount of attention in the research community. Understanding dynamic environments is seen as one of the key challenges in order to enable autonomous navigation in real-world scenarios. However, representing the temporal dimension is a challenge yet to be solved. In this paper we introduce a spatial representation, which encapsulates the statistical dynamic behavior observed in the environment. The proposed Conditional Transition Map (CTMap) is a grid-based representation that associates a probability distribution for an object exiting the cell, given its entry direction. The transition parameters are learned from a temporal signal of occupancy on cells by using a local-neighborhood cross-correlation method. In this paper, we introduce the CTMap, the learning approach and present a proof-of-concept method for estimating future paths of dynamic objects, called Conditional Probability Propagation Tree (CPPTree). The evaluation is done using a real-world dataset collected at a busy roundabout.
---
paper_title: Context-Aware Trajectory Prediction

paper_content:

Human motion and behaviour in crowded spaces is influenced by several factors, such as the dynamics of other moving agents in the scene, as well as the static elements that might be perceived as points of attraction or obstacles. In this work, we present a new model for human trajectory prediction which is able to take advantage of both human-human and human-space interactions. The future trajectory of humans, are generated by observing their past positions and interactions with the surroundings. To this end, we propose a"context-aware"recurrent neural network LSTM model, which can learn and predict human motion in crowded spaces such as a sidewalk, a museum or a shopping mall. We evaluate our model on a public pedestrian datasets, and we contribute a new challenging dataset that collects videos of humans that navigate in a (real) crowded space such as a big museum. Results show that our approach can predict human trajectories better when compared to previous state-of-the-art forecasting models.
---
paper_title: Stochastic prediction of human motion and control of robots in the service of human

paper_content:

The authors propose a control model for human cooperative robots. According to human motion which is measured by a human recognition system, future human position is predicted. Robot trajectories are changed by acceleration/deceleration/stop so as to minimize danger, which is computed by the predicted data. In this paper, a model of a stochastic process is adopted for the prediction because it can prevent inappropriate robot motion which might be caused by prediction error. A prediction plane is divided into square cells. Each cell has probability data by which cell number, direction and speed change. They are determined by the characteristics of motion of places beforehand. Computation of the stochastic state transition chain predicts the probability that a human exists in a cell at future time. Application to a laboratory room demonstrated that this method is effective for safe motion of human cooperative robots. >
---
paper_title: People tracking with human motion predictions from social forces

paper_content:

For many tasks in populated environments, robots need to keep track of current and future motion states of people. Most approaches to people tracking make weak assumptions on human motion such as constant velocity or acceleration. But even over a short period, human behavior is more complex and influenced by factors such as the intended goal, other people, objects in the environment, and social rules. This motivates the use of more sophisticated motion models for people tracking especially since humans frequently undergo lengthy occlusion events. In this paper, we consider computational models developed in the cognitive and social science communities that describe individual and collective pedestrian dynamics for tasks such as crowd behavior analysis. In particular, we integrate a model based on a social force concept into a multi-hypothesis target tracker. We show how the refined motion predictions translate into more informed probability distributions over hypotheses and finally into a more robust tracking behavior and better occlusion handling. In experiments in indoor and outdoor environments with data from a laser range finder, the social force model leads to more accurate tracking with up to two times fewer data association errors.
---
paper_title: Unfreezing the robot: Navigation in dense, interacting crowds

paper_content:

In this paper, we study the safe navigation of a mobile robot through crowds of dynamic agents with uncertain trajectories. Existing algorithms suffer from the “freezing robot” problem: once the environment surpasses a certain level of complexity, the planner decides that all forward paths are unsafe, and the robot freezes in place (or performs unnecessary maneuvers) to avoid collisions. Since a feasible path typically exists, this behavior is suboptimal. Existing approaches have focused on reducing the predictive uncertainty for individual agents by employing more informed models or heuristically limiting the predictive covariance to prevent this overcautious behavior. In this work, we demonstrate that both the individual prediction and the predictive uncertainty have little to do with the frozen robot problem. Our key insight is that dynamic agents solve the frozen robot problem by engaging in “joint collision avoidance”: They cooperatively make room to create feasible trajectories. We develop IGP, a nonparametric statistical model based on dependent output Gaussian processes that can estimate crowd interaction from data. Our model naturally captures the non-Markov nature of agent trajectories, as well as their goal-driven navigation. We then show how planning in this model can be efficiently implemented using particle based inference. Lastly, we evaluate our model on a dataset of pedestrians entering and leaving a building, first comparing the model with actual pedestrians, and find that the algorithm either outperforms human pedestrians or performs very similarly to the pedestrians. We also present an experiment where a covariance reduction method results in highly overcautious behavior, while our model performs desirably.
---
paper_title: Understanding interactions between traffic participants based on learned behaviors

paper_content:

Predicting vehicles' behaviors in a traffic scene can be very challenging due to many influences. Especially interactions with other traffic participants like vehicles or pedestrians are very crucial for the future movement while they are hard to model even with expert knowledge.
---
paper_title: Motion planning based on learning models of pedestrian and driver behaviors

paper_content:

Autonomous driving has shown the capability of providing driver convenience and enhancing safety. While introducing autonomous driving into our current traffic system, one significant issue is to make the autonomous vehicle be able to react in the same way as real human drivers. In order to ensure that an autonomous vehicle of the future will perform like human drivers, this paper proposes a vehicle motion planning model, which can represent how drivers control vehicles based on the assessment of traffic environments in the real signalized intersection. The proposed motion planning model comprises functions of pedestrian intention detection, gap detection and vehicle dynamic control. The three functions are constructed based on the analysis of actual data collected from real traffic environments. Finally, this paper demonstrates the performance of the proposed method by comparing the behaviors of our model with the behaviors of real pedestrians and human drivers. The experimental results show that our proposed model can achieve 85% recognition rate for the pedestrian crossing intention. Moreover, the vehicle controlled by the proposed motion planning model and the actual human-driven vehicle are highly similar with respect to the gap acceptance in intersections.
---
paper_title: Cyclist Trajectory Prediction Using Bidirectional Recurrent Neural Networks

paper_content:

Predicting a long-term horizon of vulnerable road users’ trajectories such as cyclists become an inevitable task for a reliable operation of highly and fully automated vehicles. In the literature, this problem is often tackled using linear dynamics-based approaches based on recursive Bayesian filters. These approaches are usually challenged when it comes to predicting long-term horizon of trajectories (more than 1 sec). Additionally, they also have difficulties in predicting non-linear motions such as maneuvers done by cyclists in traffic environments. In this work, we are proposing two novel models based on deep stacked recurrent neural networks for the task of cyclists trajectories prediction to overcome some of the aforementioned challenges. Our proposed predictive models have achieved robust prediction results when evaluated on a real-life cyclist trajectories dataset collected using vehicle-based sensors in the urban traffic environment. Furthermore, our proposed models have outperformed other traditional approaches with an improvement of more than 50% in mean error score averaged over all the predicted cyclists’ trajectories.
---
paper_title: Context-Based Path Prediction for Targets with Switching Dynamics

paper_content:

Anticipating future situations from streaming sensor data is a key perception challenge for mobile robotics and automated vehicles. We address the problem of predicting the path of objects with multiple dynamic modes. The dynamics of such targets can be described by a Switching Linear Dynamical System (SLDS). However, predictions from this probabilistic model cannot anticipate when a change in dynamic mode will occur. We propose to extract various types of cues with computer vision to provide context on the target’s behavior, and incorporate these in a Dynamic Bayesian Network (DBN). The DBN extends the SLDS by conditioning the mode transition probabilities on additional context states. We describe efficient online inference in this DBN for probabilistic path prediction, accounting for uncertainty in both measurements and target behavior. Our approach is illustrated on two scenarios in the Intelligent Vehicles domain concerning pedestrians and cyclists, so-called Vulnerable Road Users (VRUs). Here, context cues include the static environment of the VRU, its dynamic environment, and its observed actions. Experiments using stereo vision data from a moving vehicle demonstrate that the proposed approach results in more accurate path prediction than SLDS at the relevant short time horizon (1 s). It slightly outperforms a computationally more demanding state-of-the-art method.
---
paper_title: A Controlled Interactive Multiple Model Filter for Combined Pedestrian Intention Recognition and Path Prediction

paper_content:

We present a novel approach combining pedestrian intention recognition and path prediction for advanced video-based driver assistance systems. The core algorithm uses an Interacting Multiple Model Filter in combination with a Latent-dynamic Conditional Random Field model. The model integrates pedestrian dynamics and situational awareness using observations from a stereo-video system for pedestrian detection and human head pose estimation. Evaluation of our method is performed on a public available dataset addressing scenarios of lateral approaching pedestrians that might cross the road, turn into the road or stop at the curbside. During experiments, we demonstrate that the proposed approach leads to better path prediction performance in terms of a smaller lateral position error compared to state-of-the-art pedestrian intention recognition and path prediction approaches. The computational costs of our approach is comparatively low and therefore can be ported easily onto a real-time system.
---
paper_title: Motion prediction of moving objects based on autoregressive model

paper_content:

In this paper, we describe a framework for predicting future positions and orientation of moving obstacles in a time-varying environment using autoregressive model (ARM) with conditional maximum likelihood estimate of the model parameters. No constraints are placed on the obstacles motion. The proposed algorithm can be used in a variety of applications, one of which is robot motion planning in time varying environments.
---
paper_title: Road Infrastructure Indicators for Trajectory Prediction

paper_content:

Safe and comfortable path planning in a dynamic urban environment is essential to an autonomous vehicle. This requires the future trajectories of all other road users in the environment of the vehicle. These trajectories are predicted through modeling the motion and behaviour of these road users. In this work we state that for efficient trajectory prediction only motion indicators are not sufficient. Therefore, we propose using a curvilinear coordinate system with curvature as road infrastructure indicators to improve motion modeling and trajectory prediction. With experiments, we show that the curvilinear coordinate system with curvature sufficiently incorporates the road structure. Furthermore, we show that a sequence-tosequence RNN model is suitable to incorporate road curvature indicators directly into the modeling and prediction.
---
paper_title: Set-Based Prediction of Pedestrians in Urban Environments Considering Formalized Traffic Rules

paper_content:

Set-based predictions can ensure the safety of planned motions, since they provide a bounded region which includes all possible future states of nondeterministic models of other traffic participants. However, while autonomous vehicles are tested in urban environments, a set-based prediction tailored to pedestrians does not exist yet. This paper addresses this problem and presents an approach for set-based predictions of pedestrians using reachability analysis. We obtain tight over-approximations of pedestrians' reachable occupancy by incorporating the dynamics of pedestrians, contextual information, and traffic rules. In addition, since pedestrians often disregard traffic rules, our constraints automatically adapt so that such behaviors are included in the prediction. Using datasets of recorded pedestrians, we validate our proposed method and demonstrate its use for evasive maneuver planning of automated vehicles.
---
paper_title: Planning-based prediction for pedestrians

paper_content:

We present a novel approach for determining robot movements that efficiently accomplish the robot's tasks while not hindering the movements of people within the environment. Our approach models the goal-directed trajectories of pedestrians using maximum entropy inverse optimal control. The advantage of this modeling approach is the generality of its learned cost function to changes in the environment and to entirely different environments. We employ the predictions of this model of pedestrian trajectories in a novel incremental planner and quantitatively show the improvement in hindrance-sensitive robot trajectory planning provided by our approach.
---
paper_title: Bi-Prediction: Pedestrian Trajectory Prediction Based on Bidirectional LSTM Classification

paper_content:

Pedestrian trajectory prediction is important in various applications such as driverless vehicles, social robots, intelligent tracking systems and space planning. Existing methods focus on analysing the influence of neighbours but ignore the effect of the intended destinations of pedestrians which also plays a key role in route planning. In this paper, we propose a novel two- stage trajectory prediction method to yield multiple prediction trajectories with different probabilities towards different destination regions in the scene. Our method, which we refer to as Bi-Prediction, uses a bidirectional LSTM architecture to automatically classify trajectories into a small number of route classes before trajectory prediction. We have evaluated our method against two baseline methods and three state-of-art methods on two benchmark datasets. Our experimental results show that the extra classification stage improves the accuracy of the predicted trajectories.
---
paper_title: Using EM to learn motion behaviors of persons with mobile robots

paper_content:

We propose a method for learning models of people's motion behaviors in indoor environments. As people move through their environments, they do not move randomly. Instead, they often engage in typical motion patterns, related to specific locations that they might be interested in approaching and specific trajectories that they might follow in doing so. Knowledge about such patterns may enable a mobile robot to develop improved people following and obstacle avoidance skills. This paper proposes an algorithm that learns collections of typical trajectories that characterize a person's motion patterns. Data, recorded by mobile robots equipped with laser-range finders, is clustered into different types of motion using the popular expectation maximization algorithm, while simultaneously learning multiple motion patterns. Experimental results, obtained using data collected in a domestic residence and in an office building, illustrate that highly predictive models of human motion patterns can be learned.
---
paper_title: Enabling Flow Awareness for Mobile Robots in Partially Observable Environments

paper_content:

Understanding the environment is a key requirement for any autonomous robot operation. There is extensive research on mapping geometric structure and perceiving objects. However, the environment is ...
---
paper_title: Pedestrian Group Behavior in a Cellular Automaton

paper_content:

The study of crowd movement is very important for planning mass events and evacuations. Although many potentially critical incidents are hardly predictable, others like clogging, can be anticipated. Studies have shown that pedestrians in social groups frequently contribute the biggest part to crowds, and that these groups have a significant impact on crowd movement. Furthermore social cooperative behaviour does not stop in emergency situations, but continues or even gets stronger. We employ a cellular automaton with attractive and repulsive potentials for the simulation of individual pedestrians. Then we explain how to integrate a concise model for social groups suitable for the basic modelling techniques. Further aspects of group behavior are discussed, namely large groups and the behavior after a group separation, and we propose a model for these. These investigations lead us to the conclusion that there is a growing need of agent based modelling when facing advanced aspects of pedestrian crowd behavior.
---
paper_title: Human Motion Prediction Under Social Grouping Constraints

paper_content:

Accurate long-term prediction of human motion in populated spaces is an important but difficult task for mobile robots and intelligent vehicles. What makes this task challenging is that human motion is influenced by a large variety of factors including the person's intention, the presence, attributes, actions, social relations and social norms of other surrounding agents, and the geometry and semantics of the environment. In this paper, we consider the problem of computing human motion predictions that account for such factors. We formulate the task as an MDP planning problem with stochastic policies and propose a weighted random walk algorithm in which each agent is locally influenced by social forces from other nearby agents. The novelty of this paper is that we incorporate social grouping information into the prediction process reflecting the soft formation constraints that groups typically impose to their members' motion. We show that our method makes more accurate predictions than three state-of-the-art methods in terms of probabilistic and geometrical performance metrics.
---
paper_title: Long-term vehicle motion prediction

paper_content:

Future driver assistance systems will have to cope with complex traffic situations, especially in the road crossing scenario. To detect potentially hazardous situations as early as possible, it is therefore desirable to know the position and motion of the ego-vehicle and vehicles around it for several seconds in advance. For this purpose, we propose in this study a long-term prediction approach based on a combined trajectory classification and particle filter framework. As a measure for the similarity between trajectories, we introduce the quaternion-based rotationally invariant longest common subsequence (QRLCS) metric. The trajectories are classified by a radial basis function (RBF) classifier with an architecture that is able to process trajectories of arbitrary non-uniform length. The particle filter framework simultaneously tracks and assesses a large number of motion hypotheses (∼102), where the class-specific probabilities estimated by the RBF classifier are used as a-priori probabilities for the hypotheses of the particle filter. The hypotheses are clustered with a mean-shift technique and are assigned a likelihood value. Motion prediction is performed based on the cluster centre with the highest likelihood. While traditional motion prediction based on curve radius and acceleration is inaccurate especially during turning manoeuvres, we show that our approach achieves a reasonable motion prediction even for long prediction intervals of 3 s for these complex motion patterns.
---
paper_title: Natural vision based method for predicting pedestrian behaviour in urban environments

paper_content:

This paper proposes to model pedestrian behaviour in urban scenes by combining the principles of urban planning and the sociological concept of Natural Vision. This model assumes that the environment perceived by pedestrians is composed of multiple potential fields that influence their behaviour. These fields are derived from static scene elements like side-walks, cross-walks, buildings, shops entrances and dynamic obstacles like cars and buses for instance. Using this model, autonomous cars increase their level of situational awareness in the local urban space, with the ability to infer probable pedestrian paths in the scene to predict, for example, legal and illegal crossings.
---
paper_title: Improving data association by joint modeling of pedestrian trajectories and groupings

paper_content:

We consider the problem of data association in a multiperson tracking context. In semi-crowded environments, people are still discernible as individually moving entities, that undergo many interactions with other people in their direct surrounding. Finding the correct association is therefore difficult, but higher-order social factors, such as group membership, are expected to ease the problem. However, estimating group membership is a chicken-and-egg problem: knowing pedestrian trajectories, it is rather easy to find out possible groupings in the data, but in crowded scenes, it is often difficult to estimate closely interacting trajectories without further knowledge about groups. To this end, we propose a third-order graphical model that is able to jointly estimate correct trajectories and group memberships over a short time window. A set of experiments on challenging data underline the importance of joint reasoning for data association in crowded scenarios.
---
paper_title: Social force model with explicit collision prediction

paper_content:

We introduce a new specification of the social force model in which pedestrians explicitly predict the place and time of the next collision in order to avoid it. This and other specifications of the social force model are calibrated, using genetic algorithms, on a set of pedestrian trajectories, obtained tracking with laser range finders the movement of pedestrians in controlled experiments, and their performance is compared. The results show that the proposed method has a better performance in describing the trajectory set.
---
paper_title: Using road topology to improve cyclist path prediction

paper_content:

We learn motion models for cyclist path prediction on real-world tracks obtained from a moving vehicle, and propose to exploit the local road topology to obtain better predictive distributions. The tracks are extracted from the Tsinghua-Daimler Cyclist Benchmark for cyclist detection, and corrected for vehicle egomotion. Tracks are then spatially aligned to local curves and crossings in the road. We study a standard approach for path prediction in the literature based on Kalman Filters, as well as a mixture of specialized filters related to specific road orientations at junctions. Our experiments demonstrate an improved prediction accuracy (up to 20% on sharp turns) of mixing specialized motion models for canonical directions, and prior knowledge on the road topology. The new track data complements the existing video, disparity and annotation data of the original benchmark, and will be made publicly available.
---
paper_title: Unsupervised robot learning to predict person motion

paper_content:

Socially interacting robots will need to understand the intentions and recognize the behaviors of people they come in contact with. In this paper we look at how a robot can learn to recognize and predict people's intended path based on its own observations of people over time. Our approach uses people tracking on the robot from either RGBD cameras or LIDAR. The tracks are separated into homogeneous motion classes using a pre-trained SVM. Then the individual classes are clustered and prototypes are extracted from each cluster. These are then used to predict a person's future motion based on matching to a partial prototype and using the rest of the prototype as the predicted motion. Results from experiments in a kitchen environment in our lab demonstrate the capabilities of the proposed method.
---
paper_title: Probabilistic vehicle trajectory prediction over occupancy grid map via recurrent neural network

paper_content:

In this paper, we propose an efficient vehicle trajectory prediction framework based on recurrent neural network. Basically, the characteristic of the vehicle's trajectory is different from that of regular moving objects since it is affected by various latent factors including road structure, traffic rules, and driver's intention. Previous state of the art approaches use sophisticated vehicle behavior model describing these factors and derive the complex trajectory prediction algorithm, which requires a system designer to conduct intensive model optimization for practical use. Our approach is data-driven and simple to use in that it learns complex behavior of the vehicles from the massive amount of trajectory data through deep neural network model. The proposed trajectory prediction method employs the recurrent neural network called long short-term memory (LSTM) to analyze the temporal behavior and predict the future coordinate of the surrounding vehicles. The proposed scheme feeds the sequence of vehicles' coordinates obtained from sensor measurements to the LSTM and produces the probabilistic information on the future location of the vehicles over occupancy grid map. The experiments conducted using the data collected from highway driving show that the proposed method can produce reasonably good estimate of future trajectory.
---
paper_title: Intent-aware long-term prediction of pedestrian motion

paper_content:

We present a method to predict long-term motion of pedestrians, modeling their behavior as jump-Markov processes with their goal a hidden variable. Assuming approximately rational behavior, and incorporating environmental constraints and biases, including time-varying ones imposed by traffic lights, we model intent as a policy in a Markov decision process framework. We infer pedestrian state using a Rao-Blackwellized filter, and intent by planning according to a stochastic policy, reflecting individual preferences in aiming at the same goal.
---
paper_title: Learning intentions for improved human motion prediction

paper_content:

For many tasks, robots need to operate in human populated environments. Human motion prediction therefore is gaining importance. The concept of social forces defines virtual repelling and attracting forces from and to obstacles and points of interest. These social forces can then be used to model typical human movements given an environment and a person's intention. This work shows how such models can exploit typical motion patterns summarized by growing hidden Markov models (GHMMs) that can be learned from data online and without human intervention. An extensive series of experiments shows that exploiting the intended position estimated using a GHMM within a social forces based motion model yields a significant performance gain in comparison with the standard constant velocity-based models.
---
paper_title: Modelling subgroup behaviour in crowd dynamics DEM simulation

paper_content:

This paper investigates the behaviour of subgroups in crowd dynamics by means of filming and observation. An existing crowd modelling program, CrowdDMX, based on a discrete element model (DEM) has been modified on the basis of observations made in this paper and literature. Each person is represented as three overlapping circles and motion is modelled in a Newtonian manner. It incorporates psychological forces as well as physical forces in a 2D time-stepping environment. The DEM model was modified to include realistic subgroup behaviour, representing people in the crowd desiring to stay together (families, friends, etc.). Subgroup psychological forces were incorporated. The previous model only simulated individuals moving independently, which was unrealistic in some situations as shown by the observation and filming part of the study. The revised program models subgroups realistically including the tendency to avoid subgroup division in cases of contra-flow.
---
paper_title: Multimodal Interaction-aware Motion Prediction for Autonomous Street Crossing

paper_content:

For mobile robots navigating on sidewalks, it is essential to be able to safely cross street intersections. Most existing approaches rely on the recognition of the traffic light signal to make an informed crossing decision. Although these approaches have been crucial enablers for urban navigation, the capabilities of robots employing such approaches are still limited to navigating only on streets containing signalized intersections. In this paper, we address this challenge and propose a multimodal convolutional neural network framework to predict the safety of a street intersection for crossing. Our architecture consists of two subnetworks; an interaction-aware trajectory estimation stream IA-TCNN, that predicts the future states of all observed traffic participants in the scene, and a traffic light recognition stream AtteNet. Our IA-TCNN utilizes dilated causal convolutions to model the behavior of the observable dynamic agents in the scene without explicitly assigning priorities to the interactions among them. While AtteNet utilizes Squeeze-Excitation blocks to learn a content-aware mechanism for selecting the relevant features from the data, thereby improving the noise robustness. Learned representations from the traffic light recognition stream are fused with the estimated trajectories from the motion prediction stream to learn the crossing decision. Furthermore, we extend our previously introduced Freiburg Street Crossing dataset with sequences captured at different types of intersections, demonstrating complex interactions among the traffic participants. Extensive experimental evaluations on public benchmark datasets and our proposed dataset demonstrate that our network achieves state-of-the-art performance for each of the subtasks, as well as for the crossing safety prediction.
---
paper_title: Forecasting Interactive Dynamics of Pedestrians with Fictitious Play

paper_content:

We develop predictive models of pedestrian dynamics by encoding the coupled nature of multi-pedestrian interaction using game theory, and deep learning-based visual analysis to estimate person-specific behavior parameters. Building predictive models for multi-pedestrian interactions however, is very challenging due to two reasons: (1) the dynamics of interaction are complex interdependent processes, where the predicted behavior of one pedestrian can affect the actions taken by others and (2) dynamics are variable depending on an individuals physical characteristics (e.g., an older person may walk slowly while the younger person may walk faster). To address these challenges, we (1) utilize concepts from game theory to model the interdependent decision making process of multiple pedestrians and (2) use visual classifiers to learn a mapping from pedestrian appearance to behavior parameters. We evaluate our proposed model on several public multiple pedestrian interaction video datasets. Results show that our strategic planning model explains human interactions 25% better when compared to state-of-the-art methods.
---
paper_title: Learning to predict trajectories of cooperatively navigating agents

paper_content:

The problem of modeling the navigation behavior of multiple interacting agents arises in different areas including robotics, computer graphics, and behavioral science. In this paper, we present an approach to learn the composite navigation behavior of interacting agents from demonstrations. The decision process that ultimately leads to the observed continuous trajectories of the agents often also comprises discrete decisions, which partition the space of composite trajectories into homotopy classes. Therefore, our method uses a mixture probability distribution that consists of a discrete distribution over the homotopy classes and continuous distributions over the composite trajectories for each homotopy class. Our approach learns the model parameters of this distribution that match, in expectation, the observed behavior in terms of user-defined features. To compute the feature expectations over the highdimensional continuous distributions, we use Hamiltonian Markov chain Monte Carlo sampling. We exploit that the distributions are highly structured due to physical constraints and guide the sampling process to regions of high probability. We apply our approach to learning the behavior of pedestrians and demonstrate that it outperforms state-of-the-art methods.
---
paper_title: Who are you with and where are you going?

paper_content:

We propose an agent-based behavioral model of pedestrians to improve tracking performance in realistic scenarios. In this model, we view pedestrians as decision-making agents who consider a plethora of personal, social, and environmental factors to decide where to go next. We formulate prediction of pedestrian behavior as an energy minimization on this model. Two of our main contributions are simple, yet effective estimates of pedestrian destination and social relationships (groups). Our final contribution is to incorporate these hidden properties into an energy formulation that results in accurate behavioral prediction. We evaluate both our estimates of destination and grouping, as well as our accuracy at prediction and tracking against state of the art behavioral model and show improvements, especially in the challenging observational situation of infrequent appearance observations–something that might occur in thousands of webcams available on the Internet.
---
paper_title: Learning spatio-temporal patterns for predicting object behaviour

paper_content:

Abstract Rule-based systems employed to model complex object behaviours, do not necessarily provide a realistic portrayal of true behaviour. To capture the real characteristics in a specific environment, a better model may be learnt from observation. This paper presents a novel approach to learning long-term spatio-temporal patterns of objects in image sequences, using a neural network paradigm to predict future behaviour. The results demonstrate the application of our approach to the problem of predicting animal behaviour in response to a predator.
---
paper_title: Long-term Path Prediction in Urban Scenarios using Circular Distributions

paper_content:

Abstract Human ability to foresee the near future plays a key role in everyone's life to prevent potentially dangerous situations. To be able to make predictions is crucial when people have to interact with the surrounding environment. Modeling such capability can lead to the design of automated warning systems and provide moving robots with an intelligent way of interaction with changing situation. In this work we focus on a typical urban human-scene where we aim at predicting an agent's behavior using a stochastic model. In this approach, we fuse the various factors that would contribute to a human motion in different contexts. Our method uses previously observed trajectories to build point-wise circular distributions that after combination, provide a statistical smooth prediction towards the most likely areas. More specifically, a ray-launching procedure, based on a semantic segmentation, gives a coarse scene representation for collision avoidance; a nearly-constant velocity dynamic model smooths the acceleration progression and knowledge of the agent's destination may further steer the path prediction. Experimental results in structured scenes, validate the effectiveness of the method in predicting paths in comparison to actual trajectories.
---
paper_title: A Game-Theoretic Approach to Replanning-Aware Interactive Scene Prediction and Planning

paper_content:

This paper presents a novel cooperative-driving prediction and planning framework for dynamic environments based on the methods of game theory. The proposed algorithm can be used for highly automated driving on highways or as a sophisticated prediction module for advanced driver-assistance systems with no need for intervehicle communication. The main contribution of this paper is a model-based interaction-aware motion prediction of all vehicles in a scene. In contrast to other state-of-the-art approaches, the system also models the replanning capabilities of all drivers. With that, the driving strategy is able to capture complex interactions between vehicles, thus planning maneuver sequences over longer time horizons. It also enables an accurate prediction of traffic for the next immediate time step. The prediction model is supported by an interpretation of what other drivers intend to do, how they interact with traffic, and the ongoing observation. As part of the prediction loop, the proposed planning strategy incorporates the expected reactions of all traffic participants, offering cooperative and robust driving decisions. By means of experimental results under simulated highway scenarios, the validity of the proposed concept and its real-time capability is demonstrated.
---
paper_title: Predicting trajectories of vehicles using large-scale motion priors

paper_content:

We present a simple yet effective paradigm to accurately predict the future trajectories of observed vehicles in dense city environments. We equipped a large fleet of cars with cameras and performed city-scale structure-from-motion to accurately reconstruct 10M positions of their trajectories spanning over 1000h of driving.We demonstrate that this information can be used as a powerful high-fidelity prior to predict future trajectories of newly observed vehicles in the area without the need for any knowledge of road infrastructure or vehicle motion models. By relating the current position of the observed car to a large dataset of the previously exhibited motion in the area we can directly perform prediction of its future position.We evaluate our method on two large-scale data sets from San Francisco and New York City and demonstrate an order of magnitude improvement compared to a linear-motion based method. We also demonstrate that the performance naturally improves with the amount of data and ultimately yields a system that can accurately predict vehicle motion in challenging situations across extremes in traffic, time, and weather conditions.
---
paper_title: Pedestrian's Trajectory Forecast in Public Traffic with Artificial Neural Networks

paper_content:

This paper focuses on forecasting of pedestrian's short-time trajectories up to 2.5 s for traffic safety applications. We present a self-learning approach based on artificial neural network movement models and compare it to traditional constant velocity Kalman Filter prediction and extrapolation of polynomials fitted using a least-squares error. Trajectories of uninstructed pedestrians in public traffic at a real urban intersection are acquired by a wide angle stereo camera setup in combination with a 3D head tracking framework. Results using this real-world data show that the artificial neural network significantly improves forecast quality compared to other approaches especially for critical traffic scenes including velocity changes such as starting and stopping. For those velocity changes a reduction of position estimation errors of about 21% compared to the Kalman Filter and to extrapolation of polynomials is obtained. By means of a concrete pedestrian-vehicle scenario we demonstrate the benefit of the proposed approach for an advanced driver assistant system in terms of reaction time.
---
paper_title: Learning to navigate through crowded environments

paper_content:

The goal of this research is to enable mobile robots to navigate through crowded environments such as indoor shopping malls, airports, or downtown side walks. The key research question addressed in this paper is how to learn planners that generate human-like motion behavior. Our approach uses inverse reinforcement learning (IRL) to learn human-like navigation behavior based on example paths. Since robots have only limited sensing, we extend existing IRL methods to the case of partially observable environments. We demonstrate the capabilities of our approach using a realistic crowd flow simulator in which we modeled multiple scenarios in crowded environments. We show that our planner learned to guide the robot along the flow of people when the environment is crowded, and along the shortest path if no people are around.
---
paper_title: Modeling cooperative navigation in dense human crowds

paper_content:

For robots to be a part of our daily life, they need to be able to navigate among crowds not only safely but also in a socially compliant fashion. This is a challenging problem because humans tend to navigate by implicitly cooperating with one another to avoid collisions, while heading toward their respective destinations. Previous approaches have used handcrafted functions based on proximity to model human-human and human-robot interactions. However, these approaches can only model simple interactions and fail to generalize for complex crowded settings. In this paper, we develop an approach that models the joint distribution over future trajectories of all interacting agents in the crowd, through a local interaction model that we train using real human trajectory data. The interaction model infers the velocity of each agent based on the spatial orientation of other agents in his vicinity. During prediction, our approach infers the goal of the agent from its past trajectory and uses the learned model to predict its future trajectory. We demonstrate the performance of our method against a state-of-the-art approach on a public dataset and show that our model outperforms when predicting future trajectories for longer horizons.
---
paper_title: Recognition of situation classes at road intersections

paper_content:

The recognition and prediction of situations is an indispensable skill of future driver assistance systems. This study focuses on the recognition of situations involving two vehicles at intersections. For each vehicle, a set of possible future motion trajectories is estimated and rated based on a motion database for a time interval of 2–4 seconds ahead. Realistic situations are generated by a pairwise combination of these individual motion trajectories and classified according to nine categories with a polynomial classifier. In the proposed framework, situations are penalised for which the time to collision significantly exceeds the typical human reaction time. The correspondingly favoured situations are combined by a probabilistic framework, resulting in a more reliable situation recognition and collision detection than obtained based on independent motion hypotheses. The proposed method is evaluated on a real-world differential GPS data set acquired during a test drive of 10 km, including three road intersections. Our method is typically able to recognise the situation correctly about 1–2 seconds before the distance to the intersection centre becomes minimal.
---
paper_title: Estimation of Multivehicle Dynamics by Considering Contextual Information

paper_content:

Human drivers are endowed with an inborn ability to put themselves in the position of other drivers and reason about their behavior and intended actions. State-of-the-art driving-assistance systems, on the other hand, are generally limited to physical models and ad hoc safety rules. In order to drive safely amongst humans, autonomous vehicles need to develop an understanding of the situation in the form of a high-level description of the state of traffic participants. This paper presents a probabilistic model to estimate the state of vehicles by considering interactions between drivers immersed in traffic. The model is defined within a probabilistic filtering framework; estimation and prediction are carried out with statistical inference techniques. Memory requirements increase linearly with the number of vehicles, and thus, it is possible to scale the model to complex scenarios involving many participants. The approach is validated using real-world data collected by a group of interacting ground vehicles.
---
paper_title: Context-Aware Pedestrian Motion Prediction In Urban Intersections

paper_content:

This paper presents a novel context-based approach for pedestrian motion prediction in crowded, urban intersections, with the additional flexibility of prediction in similar, but new, environments. Previously, Chen et. al. combined Markovian-based and clustering-based approaches to learn motion primitives in a grid-based world and subsequently predict pedestrian trajectories by modeling the transition between learned primitives as a Gaussian Process (GP). This work extends that prior approach by incorporating semantic features from the environment (relative distance to curbside and status of pedestrian traffic lights) in the GP formulation for more accurate predictions of pedestrian trajectories over the same timescale. We evaluate the new approach on real-world data collected using one of the vehicles in the MIT Mobility On Demand fleet. The results show 12.5% improvement in prediction accuracy and a 2.65 times reduction in Area Under the Curve (AUC), which is used as a metric to quantify the span of predicted set of trajectories, such that a lower AUC corresponds to a higher level of confidence in the future direction of pedestrian motion.
---
paper_title: Will the Pedestrian Cross? A Study on Pedestrian Path Prediction

paper_content:

Future vehicle systems for active pedestrian safety will not only require a high recognition performance but also an accurate analysis of the developing traffic situation. In this paper, we present a study on pedestrian path prediction and action classification at short subsecond time intervals. We consider four representative approaches: two novel approaches (based on Gaussian process dynamical models and probabilistic hierarchical trajectory matching) that use augmented features derived from dense optical flow and two approaches as baseline that use positional information only (a Kalman filter and its extension to interacting multiple models). In experiments using stereo vision data obtained from a vehicle, we investigate the accuracy of path prediction and action classification at various time horizons, the effect of various errors (image localization, vehicle egomotion estimation), and the benefit of the proposed approaches. The scenario of interest is that of a crossing pedestrian, who might stop or continue walking at the road curbside. Results indicate similar performance of the four approaches on walking motion, with near-linear dynamics. During stopping, however, the two newly proposed approaches, with nonlinear and/or higher order models and augmented motion features, achieve a more accurate position prediction of 10-50 cm at a time horizon of 0-0.77 s around the stopping event.
---
paper_title: Intention-aware online POMDP planning for autonomous driving in a crowd

paper_content:

This paper presents an intention-aware online planning approach for autonomous driving amid many pedestrians. To drive near pedestrians safely, efficiently, and smoothly, autonomous vehicles must estimate unknown pedestrian intentions and hedge against the uncertainty in intention estimates in order to choose actions that are effective and robust. A key feature of our approach is to use the partially observable Markov decision process (POMDP) for systematic, robust decision making under uncertainty. Although there are concerns about the potentially high computational complexity of POMDP planning, experiments show that our POMDP-based planner runs in near real time, at 3 Hz, on a robot golf cart in a complex, dynamic environment. This indicates that POMDP planning is improving fast in computational efficiency and becoming increasingly practical as a tool for robot planning under uncertainty. © 2015 IEEE
---
paper_title: Motion Prediction of Traffic Actors for Autonomous Driving using Deep Convolutional Networks.

paper_content:

Recent algorithmic improvements and hardware breakthroughs resulted in a number of success stories in the field of AI impacting our daily lives. However, despite its ubiquity AI is only just starting to make advances in what may arguably have the largest impact thus far, the nascent field of autonomous driving. In this work we discuss this important topic and address one of crucial aspects of the emerging area, the problem of predicting future state of autonomous vehicle's surrounding necessary for safe and efficient operations. We introduce a deep learning-based approach that takes into account current state of traffic actors and produces rasterized representations of each actor's vicinity. The raster images are then used by deep convolutional models to infer future movement of actors while accounting for inherent uncertainty of the prediction task. Extensive experiments on real-world data strongly suggest benefits of the proposed approach. Moreover, following successful tests the system was deployed to a fleet of autonomous vehicles.
---
paper_title: Multiple target tracking in world coordinate with single, minimally calibrated camera

paper_content:

Tracking multiple objects is important in many application domains. We propose a novel algorithm for multi-object tracking that is capable of working under very challenging conditions such as minimal hardware equipment, uncalibrated monocular camera, occlusions and severe background clutter. To address this problem we propose a new method that jointly estimates object tracks, estimates corresponding 2D/3D temporal trajectories in the camera reference system as well as estimates the model parameters (pose, focal length, etc) within a coherent probabilistic formulation. Since our goal is to estimate stable and robust tracks that can be univocally associated to the object IDs, we propose to include in our formulation an interaction (attraction and repulsion) model that is able to model multiple 2D/3D trajectories in space-time and handle situations where objects occlude each other. We use a MCMC particle filtering algorithm for parameter inference and propose a solution that enables accurate and efficient tracking and camera model estimation. Qualitative and quantitative experimental results obtained using our own dataset and the publicly available ETH dataset shows very promising tracking and camera estimation results.
---
paper_title: Pedestrian Reactive Navigation for Crowd Simulation: a Predictive Approach Abstract

paper_content:

This paper addresses the problem of virtual pedestrian autonomous navigation for crowd simulation. It describes a method for solving interactions between pedestrians and avoiding inter-collisions. Our approach is agent-based and predictive: each agent perceives surrounding agents and extrapolates their trajectory in order to react to potential collisions. We aim at obtaining realistic results, thus the proposed model is calibrated from experimental motion capture data. Our method is shown to be valid and solves major drawbacks compared to previous approaches such as oscillations due to a lack of anticipation. We first describe the mathematical representation used in our model, we then detail its implementation, and finally, its calibration and validation from real data.
---
paper_title: Pedestrian Trajectory Prediction in Extremely Crowded Scenarios

paper_content:

Pedestrian trajectory prediction under crowded circumstances is a challenging problem owing to human interaction and the complexity of the trajectory pattern. Various methods have been proposed for solving this problem, ranging from traditional Bayesian analysis to Social Force model and deep learning methods. However, most existing models heavily depend on specific scenarios because the trajectory model is constructed in absolute coordinates even though the motion trajectory as well as human interaction are in relative motion. In this study, a novel trajectory prediction model is proposed to capture the relative motion of pedestrians in extremely crowded scenarios. Trajectory sequences and human interaction are first represented with relative motion and then integrated to our model to predict pedestrians’ trajectories. The proposed model is based on Long Short Term Memory (LSTM) structure and consists of an encoder and a decoder which are trained by truncated back propagation. In addition, an anisotropic neighborhood setting is proposed instead of traditional neighborhood analysis. The proposed approach is validated using trajectory data acquired at an extremely crowded train station in Tokyo, Japan. The trajectory prediction experiments demonstrated that the proposed method outperforms existing methods and is stable for predictions of varying length even when the model is trained with a controlled short trajectory sequence.
---
paper_title: Simulating and Evaluating the Local Behavior of Small Pedestrian Groups

paper_content:

Recent advancements in local methods have significantly improved the collision avoidance behavior of virtual characters. However, existing methods fail to take into account that in real life pedestrians tend to walk in small groups, consisting mainly of pairs or triples of individuals. We present a novel approach to simulate the walking behavior of such small groups. Our model describes how group members interact with each other, with other groups and individuals. We highlight the potential of our method through a wide range of test-case scenarios. We evaluate the results from our simulations using a number of quantitative quality metrics, and also provide visual and numerical comparisons with video footages of real crowds.
---
paper_title: DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents

paper_content:

We introduce a Deep Stochastic IOC RNN Encoder-decoder framework, DESIRE, for the task of future predictions of multiple interacting agents in dynamic scenes. DESIRE effectively predicts future locations of objects in multiple scenes by 1) accounting for the multi-modal nature of the future prediction (i.e., given the same context, future may vary), 2) foreseeing the potential future outcomes and make a strategic prediction based on that, and 3) reasoning not only from the past motion history, but also from the scene context as well as the interactions among the agents. DESIRE achieves these in a single end-to-end trainable neural network model, while being computationally efficient. The model first obtains a diverse set of hypothetical future prediction samples employing a conditional variational auto-encoder, which are ranked and refined by the following RNN scoring-regression module. Samples are scored by accounting for accumulated future rewards, which enables better long-term strategic decisions similar to IOC frameworks. An RNN scene context fusion module jointly captures past motion histories, the semantic scene context and interactions among multiple agents. A feedback mechanism iterates over the ranking and refinement to further boost the prediction accuracy. We evaluate our model on two publicly available datasets: KITTI and Stanford Drone Dataset. Our experiments show that the proposed model significantly improves the prediction accuracy compared to other baseline methods.
---
paper_title: Augmented dictionary learning for motion prediction

paper_content:

Developing accurate models and efficient representations of multivariate trajectories is important for understanding the behavior patterns of mobile agents. This work presents a dictionary learning algorithm for developing a part-based trajectory representation, which combines merits of the existing Markovian-based and clustering-based approaches. In particular, this work presents the augmented semi-nonnegative sparse coding (ASNSC) algorithm for solving a constrained dictionary learning problem, and shows that the proposed method would converge to a local optimum given a convexity condition. We consider a trajectory modeling application, in which the learned dictionary atoms correspond to local motion patterns. Classical semi-nonnegative sparse coding approaches would add dictionary atoms with opposite signs to reduce the representational error, which can lead to learning noisy dictionary atoms that correspond poorly to local motion patterns. ASNSC addresses this problem and learns a concise set of intuitive motion patterns. ASNSC shows significant improvement over existing trajectory modeling methods in both prediction accuracy and computational time, as revealed by extensive numerical analysis on real datasets.
---
paper_title: The Walking Behaviour of Pedestrian Social Groups and Its Impact on Crowd Dynamics

paper_content:

Human crowd motion is mainly driven by self-organized processes based on local interactions among pedestrians. While most studies of crowd behaviour consider only interactions among isolated individuals, it turns out that up to 70% of people in a crowd are actually moving in groups, such as friends, couples, or families walking together. These groups constitute medium-scale aggregated structures and their impact on crowd dynamics is still largely unknown. In this work, we analyze the motion of approximately 1500 pedestrian groups under natural condition, and show that social interactions among group members generate typical group walking patterns that influence crowd dynamics. At low density, group members tend to walk side by side, forming a line perpendicular to the walking direction. As the density increases, however, the linear walking formation is bent forward, turning it into a V-like pattern. These spatial patterns can be well described by a model based on social communication between group members. We show that the V-like walking pattern facilitates social interactions within the group, but reduces the flow because of its "non-aerodynamic" shape. Therefore, when crowd density increases, the group organization results from a trade-off between walking faster and facilitating social exchange. These insights demonstrate that crowd dynamics is not only determined by physical constraints induced by other pedestrians and the environment, but also significantly by communicative, social interactions among individuals.
---
paper_title: Location-Velocity Attention for Pedestrian Trajectory Prediction

paper_content:

Pedestrian path forecasting is crucial in applications such as smart video surveillance. It is a challenging task because of the complex crowd movement patterns in the scenes. Most of existing state-of-the-art LSTM based prediction methods require rich context like labelled static obstacles, labelled entrance/exit regions and even the background scene. Furthermore, incorporating contextual information into trajectory prediction increases the computational overhead and decreases the generalization of the prediction models across different scenes. In this paper, we propose a joint Location-Velocity Attention LSTM based method to predict trajectories. Specifically, a module is designed to tweak the LSTM network and an attention mechanism is trained to learn to optimally combine the location and the velocity information of pedestrians in the prediction process. We have evaluated our approach against other baselines and state-of-the-art methods on several publicly available datasets. The results show that it not only outperforms other prediction methods but it also has a good generalization ability.
---
paper_title: SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints

paper_content:

This paper addresses the problem of path prediction for multiple interacting agents in a scene, which is a crucial step for many autonomous platforms such as self-driving cars and social robots. We present SoPhie; an interpretable framework based on Generative Adversarial Network (GAN), which leverages two sources of information, the path history of all the agents in a scene, and the scene context information, using images of the scene. To predict a future path for an agent, both physical and social information must be leveraged. Previous work has not been successful to jointly model physical and social interactions. Our approach blends a social attention mechanism with physical attention that helps the model to learn where to look in a large scene and extract the most salient parts of the image relevant to the path. Whereas, the social attention component aggregates information across the different agent interactions and extracts the most important trajectory information from the surrounding neighbors. SoPhie also takes advantage of GAN to generates more realistic samples and to capture the uncertain nature of the future paths by modeling its distribution. All these mechanisms enable our approach to predict socially and physically plausible paths for the agents and to achieve state-of-the-art performance on several different trajectory forecasting benchmarks.
---
paper_title: A velocity-based approach for simulating human collision avoidance

paper_content:

We present a velocity-based model for realistic collision avoidance among virtual characters. Our approach is elaborated from experimental data and is based on the simple hypothesis that an individual tries to resolve collisions long in advance by slightly adapting its motion.
---
paper_title: INFER: INtermediate representations for FuturE pRediction

paper_content:

In urban driving scenarios, forecasting future trajectories of surrounding vehicles is of paramount importance. While several approaches for the problem have been proposed, the best-performing ones tend to require extremely detailed input representations (e.g. image sequences). As a result, such methods do not generalize to datasets they have not been trained on. In this paper, we propose intermediate representations that are particularly well-suited for future prediction. As opposed to using texture (color) information from images, we condition on semantics and train an autoregressive model to accurately predict future trajectories of traffic participants (vehicles) (see Fig. above). We demonstrate that semantics provide a significant boost over techniques that operate over raw pixel intensities/disparities. Uncharacteristic of state-of-the-art approaches, our representations and models generalize across different sensing modalities (stereo imagery, LiDAR, a combination of both), and also across completely different datasets, collected across several cities, and also across countries where people drive on opposite sides of the road (left-handed vs right-handed driving). Additionally, we demonstrate an application of our approach in multi-object tracking (data association). To foster further research in transferable representations and ensure reproducibility, we release all our code and data.33More qualitative and quantitative results, along with code and data can be found at https://rebrand.ly/INFER-results
---
paper_title: Probabilistic Map-based Pedestrian Motion Prediction Taking Traffic Participants into Consideration

paper_content:

As pedestrians are one of the most vulnerable traffic participants, their motion prediction is of utmost importance for intelligent transportation systems. Predicting motions of pedestrians is especially hard since they move in less structured environments and have less inertia compared to road vehicles. To account for this uncertainty, we present an approach for probabilistic prediction of pedestrian motion using Markov chains. In contrast to previous work, we not only consider motion models, constraints from a semantic map, and various goals, but also explicitly adapt the prediction based on crash probabilities with other traffic participants. Also, our approach works in any situation; this is typically challenging for pure machine learning techniques that learn behaviors for a particular road section and which might consequently struggle with a different road section. The usefulness of combining the aforementioned aspects in a single approach is demonstrated by an evaluation using recordings of real pedestrians.
---
paper_title: Patch to the Future: Unsupervised Visual Prediction

paper_content:

In this paper we present a conceptually simple but surprisingly powerful method for visual prediction which combines the effectiveness of mid-level visual elements with temporal modeling. Our framework can be learned in a completely unsupervised manner from a large collection of videos. However, more importantly, because our approach models the prediction framework on these mid-level elements, we can not only predict the possible motion in the scene but also predict visual appearances--how are appearances going to change with time. This yields a visual "hallucination" of probable events on top of the scene. We show that our method is able to accurately predict and visualize simple future events, we also show that our approach is comparable to supervised methods for event prediction.
---
paper_title: Human Motion Behaviour Aware Planner (HMBAP) for path planning in dynamic human environments

paper_content:

For a robot navigating in a human inhabited dynamic environment, the knowledge of how the robot's movement can influence the trajectory of people around it can be very valuable. In this work we present a Human Motion Behaviour Aware Planner (HMBAP) which incorporates a Human Motion Behaviour Model (HMBM) in its planning stage to take advantage of this. HMBM is an obstacle avoidance model for people based on social forces which gives the robot an understanding of how people would react to its planned path. This information is useful for the robot to avoid imminent collisions with people in constricted spaces. The resulting robot behaviour is similar to how a human would move (in terms of avoidance behaviour) in a similar situation. We believe that this is a desirable feature for a robot navigating in a human inhabited environment. Our method shows good human-like navigation behaviour in situations where past methods fail to find a solution.
---
paper_title: Building a human behavior map from local observations

paper_content:

This paper presents a novel method for classifying regions from human movements in service robots' working environments. The entire space is segmented subject to the class type according to the functionality or affordance of each place which accommodates a typical human behavior. This is achieved based on a grid map in two steps. First a probabilistic model is developed to capture human movements for each grid cell by using a non-ergodic HMM. Then the learned transition probabilities corresponding to these movements are used to cluster all cells by using the K-means algorithm. The knowledge of typical human movements for each location, represented by the prototypes from K-means and summarized in a ‘behavior-based map’, enables a robot to adjust the strategy of interacting with people according to where they are located, and thus greatly enhances its capability to assist people. The performance of the proposed classification method is demonstrated by experimental results from 8 hours of data that are collected in a kitchen environment.
---
paper_title: Human Trajectory Prediction using Spatially aware Deep Attention Models

paper_content:

Trajectory Prediction of dynamic objects is a widely studied topic in the field of artificial intelligence. Thanks to a large number of applications like predicting abnormal events, navigation system for the blind, etc. there have been many approaches to attempt learning patterns of motion directly from data using a wide variety of techniques ranging from hand-crafted features to sophisticated deep learning models for unsupervised feature learning. All these approaches have been limited by problems like inefficient features in the case of hand crafted features, large error propagation across the predicted trajectory and no information of static artefacts around the dynamic moving objects. We propose an end to end deep learning model to learn the motion patterns of humans using different navigational modes directly from data using the much popular sequence to sequence model coupled with a soft attention mechanism. We also propose a novel approach to model the static artefacts in a scene and using these to predict the dynamic trajectories. The proposed method, tested on trajectories of pedestrians, consistently outperforms previously proposed state of the art approaches on a variety of large scale data sets. We also show how our architecture can be naturally extended to handle multiple modes of movement (say pedestrians, skaters, bikers and buses) simultaneously.
---
paper_title: Multimodal Probabilistic Model-Based Planning for Human-Robot Interaction

paper_content:

This paper presents a method for constructing human-robot interaction policies in settings where multimodality, i.e., the possibility of multiple highly distinct futures, plays a critical role in decision making. We are motivated in this work by the example of traffic weaving, e.g., at highway on-ramps/off-ramps, where entering and exiting cars must swap lanes in a short distance-a challenging negotiation even for experienced drivers due to the inherent multimodal uncertainty of who will pass whom. Our approach is to learn multimodal probability distributions over future human actions from a dataset of human-human exemplars and perform real-time robot policy construction in the resulting environment model through massively parallel sampling of human responses to candidate robot action sequences. Direct learning of these distributions is made possible by recent advances in the theory of conditional variational autoencoders (CVAEs), whereby we learn action distributions simultaneously conditioned on the present interaction history, as well as candidate future robot actions in order to take into account response dynamics. We demonstrate the efficacy of this approach with a human-in-the-loop simulation of a traffic weaving scenario.
---
paper_title: Probabilistic Autonomous Robot Navigation in Dynamic Environments with Human Motion Prediction

paper_content:

This paper considers the problem of autonomous robot navigation in dynamic and congested environments. The predictive navigation paradigm is proposed where probabilistic planning is integrated with obstacle avoidance along with future motion prediction of humans and/or other obstacles. Predictive navigation is performed in a global manner with the use of a hierarchical Partially Observable Markov Decision Process (POMDP) that can be solved on-line at each time step and provides the actual actions the robot performs. Obstacle avoidance is performed within the predictive navigation model with a novel approach by deciding paths to the goal position that are not obstructed by other moving objects movement with the use of future motion prediction and by enabling the robot to increase or decrease its speed of movement or by performing detours. The robot is able to decide which obstacle avoidance behavior is optimal in each case within the unified navigation model employed.
---
paper_title: Human-robot co-navigation using anticipatory indicators of human walking motion

paper_content:

Mobile, interactive robots that operate in human-centric environments need the capability to safely and efficiently navigate around humans. This requires the ability to sense and predict human motion trajectories and to plan around them. In this paper, we present a study that supports the existence of statistically significant biomechanical turn indicators of human walking motions. Further, we demonstrate the effectiveness of these turn indicators as features in the prediction of human motion trajectories. Human motion capture data is collected with predefined goals to train and test a prediction algorithm. Use of anticipatory features results in improved performance of the prediction algorithm. Lastly, we demonstrate the closed-loop performance of the prediction algorithm using an existing algorithm for motion planning within dynamic environments. The anticipatory indicators of human walking motion can be used with different prediction and/or planning algorithms for robotics; the chosen planning and prediction algorithm demonstrates one such implementation for human-robot co-navigation.
---
paper_title: Learning Social Etiquette: Human Trajectory Understanding In Crowded Scenes

paper_content:

Humans navigate crowded spaces such as a university campus by following common sense rules based on social etiquette. In this paper, we argue that in order to enable the design of new target tracking or trajectory forecasting methods that can take full advantage of these rules, we need to have access to better data in the first place. To that end, we contribute a new large-scale dataset that collects videos of various types of targets (not just pedestrians, but also bikers, skateboarders, cars, buses, golf carts) that navigate in a real world outdoor environment such as a university campus. Moreover, we introduce a new characterization that describes the “social sensitivity” at which two targets interact. We use this characterization to define “navigation styles” and improve both forecasting models and state-of-the-art multi-target tracking–whereby the learnt forecasting models help the data association step.
---
paper_title: Group LSTM: Group Trajectory Prediction in Crowded Scenarios

paper_content:

The analysis of crowded scenes is one of the most challenging scenarios in visual surveillance, and a variety of factors need to be taken into account, such as the structure of the environments, and the presence of mutual occlusions and obstacles. Traditional prediction methods (such as RNN, LSTM, VAE, etc.) focus on anticipating individual’s future path based on the precise motion history of a pedestrian. However, since tracking algorithms are generally not reliable in highly dense scenes, these methods are not easily applicable in real environments. Nevertheless, it is very common that people (friends, couples, family members, etc.) tend to exhibit coherent motion patterns. Motivated by this phenomenon, we propose a novel approach to predict future trajectories in crowded scenes, at the group level. First, by exploiting the motion coherency, we cluster trajectories that have similar motion trends. In this way, pedestrians within the same group can be well segmented. Then, an improved social-LSTM is adopted for future path prediction. We evaluate our approach on standard crowd benchmarks (the UCY dataset and the ETH dataset), demonstrating its efficacy and applicability.
---
paper_title: Generative Multi-Agent Behavioral Cloning

paper_content:

We propose and study the problem of generative multi-agent behavioral cloning, where the goal is to learn a generative, i.e., non-deterministic, multi-agent policy from pre-collected demonstration data. Building upon advances in deep generative models, we present a hierarchical policy framework that can tractably learn complex mappings from input states to distributions over multi-agent action spaces by introducing a hierarchy with macro-intent variables that encode long-term intent. In addition to synthetic settings, we show how to instantiate our framework to effectively model complex interactions between basketball players and generate realistic multi-agent trajectories of basketball gameplay over long time periods. We validate our approach using both quantitative and qualitative evaluations, including a user study comparison conducted with professional sports analysts.
---
paper_title: Multi-Agent Tensor Fusion for Contextual Trajectory Prediction

paper_content:

Accurate prediction of others' trajectories is essential for autonomous driving. Trajectory prediction is challenging because it requires reasoning about agents' past movements, social interactions among varying numbers and kinds of agents, constraints from the scene context, and the stochasticity of human behavior. Our approach models these interactions and constraints jointly within a novel Multi-Agent Tensor Fusion (MATF) network. Specifically, the model encodes multiple agents' past trajectories and the scene context into a Multi-Agent Tensor, then applies convolutional fusion to capture multiagent interactions while retaining the spatial structure of agents and the scene context. The model decodes recurrently to multiple agents' future trajectories, using adversarial loss to learn stochastic predictions. Experiments on both highway driving and pedestrian crowd datasets show that the model achieves state-of-the-art prediction accuracy.
---
paper_title: Pedestrian Path, Pose, and Intention Prediction Through Gaussian Process Dynamical Models and Pedestrian Activity Recognition

paper_content:

According to several reports published by worldwide organizations, thousands of pedestrians die in road accidents every year. Due to this fact, vehicular technologies have been evolving with the intent of reducing these fatalities. This evolution has not finished yet, since, for instance, the predictions of pedestrian paths could improve the current automatic emergency braking systems. For this reason, this paper proposes a method to predict future pedestrian paths, poses, and intentions up to 1 s in advance. This method is based on balanced Gaussian process dynamical models (B-GPDMs), which reduce the 3-D time-related information extracted from key points or joints placed along pedestrian bodies into low-dimensional spaces. The B-GPDM is also capable of inferring future latent positions and reconstruct their associated observations. However, learning a generic model for all kinds of pedestrian activities normally provides less accurate predictions. For this reason, the proposed method obtains multiple models of four types of activity, i.e., walking, stopping, starting, and standing, and selects the most similar model to estimate future pedestrian states. This method detects starting activities 125 ms after the gait initiation with an accuracy of 80% and recognizes stopping intentions 58.33 ms before the event with an accuracy of 70%. Concerning the path prediction, the mean error for stopping activities at a time-to-event (TTE) of 1 s is 238.01 ± 206.93 mm and, for starting actions, the mean error at a TTE of 0 s is 331.93 ± 254.73 mm.
---
paper_title: Long-term Planning by Short-term Prediction

paper_content:

We consider planning problems, that often arise in autonomous driving applications, in which an agent should decide on immediate actions so as to optimize a long term objective. For example, when a car tries to merge in a roundabout it should decide on an immediate acceleration/braking command, while the long term effect of the command is the success/failure of the merge. Such problems are characterized by continuous state and action spaces, and by interaction with multiple agents, whose behavior can be adversarial. We argue that dual versions of the MDP framework (that depend on the value function and the $Q$ function) are problematic for autonomous driving applications due to the non Markovian of the natural state space representation, and due to the continuous state and action spaces. We propose to tackle the planning task by decomposing the problem into two phases: First, we apply supervised learning for predicting the near future based on the present. We require that the predictor will be differentiable with respect to the representation of the present. Second, we model a full trajectory of the agent using a recurrent neural network, where unexplained factors are modeled as (additive) input nodes. This allows us to solve the long-term planning problem using supervised learning techniques and direct optimization over the recurrent neural network. Our approach enables us to learn robust policies by incorporating adversarial elements to the environment.
---
paper_title: Learning Motion Patterns of People for Compliant Robot Motion

paper_content:

Whenever people move through their environments they do not move randomly. Instead, they usually follow specific trajectories or motion patterns corresponding to their intentions. Knowledge about such patterns enables a mobile robot to robustly keep track of persons in its environment and to improve its behavior. In this paper we propose a technique for learning collections of trajectories that characterize typical motion patterns of persons. Data recorded with laser-range finders are clustered using the expectation maximization algorithm. Based on the result of the clustering process, we derive a hidden Markov model that is applied to estimate the current and future positions of persons based on sensory input. We also describe how to incorporate the probabilistic belief about the potential trajectories of persons into the path planning process of a mobile robot. We present several experiments carried out in different environments with a mobile robot equipped with a laser-range scanner and a camera system...
---
paper_title: Novel planning-based algorithms for human motion prediction

paper_content:

Human motion prediction from visual tracking is a challenging problem with a wide array of applications such as robotics, video surveillance and situation understanding. Recently, planning-based methods -which assume that people move by planning over a cost function- have emerged as one of the most promising alternatives. Nevertheless, state of the art planning based algorithms have shortcomings regarding their computational complexity and ability to predict for arbitrary time intervals. This paper addresses these shortcomings by leveraging alternative planning techniques (Fast Marching Method) and formulating efficient algorithms for goal estimation and full spatiotemporal prediction with lower complexity than comparable approaches. In preliminary experiments, the proposed method significantly outperforms the accuracy of the current state-of-the-art approach while reducing the computation time by a factor of 30 using a parallel version of our algorithm.
---
paper_title: Monte Carlo road safety reasoning

paper_content:

This paper presents a framework for reasoning about the future motion of multiple objects in a road scene. Unlike previous approaches, we do not look for known dangerous configurations of objects, but rather we reason about the future paths of all objects in the scene, and assess their danger. Monte Carlo path planning is used to generate a probability distribution for the possible future motion of every car in the scene. This framework can be used to either control the car, or to display warnings for the driver.
---
paper_title: Pedestrian path prediction using body language traits

paper_content:

Driver Assistance Systems have achieved a high level of maturity in the latest years. As an example of that, sophisticated pedestrian protection systems are already available in a number of commercial vehicles from several OEMs. However, accurate pedestrian path prediction is needed in order to go a step further in terms of safety and reliability, since it can make the difference between effective and non-effective intervention. In this paper, we consider the three-dimensional pedestrian body language in order to perform path prediction in a probabilistic framework. For this purpose, the different body parts and joints are detected using stereo vision. We propose the use of GPDM (Gaussian Process Dynamical Models) for reducing the high dimensionality of the input feature vector (composed by joints and displacement vectors) in the 3D pose space and for learning the pedestrian dynamics in a latent space. Experimental results show that accurate path prediction can be achieved at a time horizon of ≈ 0.8 s.
---
paper_title: Modelling and Predicting Rhythmic Flow Patterns in Dynamic Environments

paper_content:

We present a time-dependent probabilistic map able to model and predict flow patterns of people in indoor environments. The proposed representation models the likelihood of motion direction on a grid-based map by a set of harmonic functions, which efficiently capture long-term (minutes to weeks) variations of crowd movements over time. The evaluation, performed on data from two real environments, shows that the proposed model enables prediction of human movement patterns in the future. Potential applications include human-aware motion planning, improving the efficiency and safety of robot navigation.
---
paper_title: Pedestrian Behavior Prediction based on Motion Patterns for Vehicle-to-Pedestrian Collision Avoidance

paper_content:

This paper proposes a prediction method for vehicle-to-pedestrian collision avoidance, which learns and then predicts pedestrian behaviors as their motion instances are being observed. During learning, known trajectories are clustered to form motion patterns (MP), which become knowledge a priori to a multi-level prediction model that predicts long-term or short-term pedestrian behaviors. Simulation results show that it works well in a complex structured environment and the prediction is consistent with actual behaviors.
---
paper_title: Predicting actions to act predictably: Cooperative partial motion planning with maximum entropy models

paper_content:

This paper reports on a data-driven motion planning approach for interaction-aware, socially-compliant robot navigation among human agents. Autonomous mobile robots navigating in workspaces shared with human agents require motion planning techniques providing seamless integration and smooth navigation in such. Smooth integration in mixed scenarios calls for two abilities of the robot: predicting actions of others and acting predictably for them. The former requirement requests trainable models of agent behaviors in order to accurately forecast their actions in the future, taking into account their reaction on the robot's decisions. A human-like navigation style of the robot facilitates other agents—most likely not aware of the underlying planning technique applied—to predict the robot motion vice versa, resulting in smoother joint navigation. The approach presented in this paper is based on a feature-based maximum entropy model and is able to guide a robot in an unstructured, real-world environment. The model is trained to predict joint behavior of heterogeneous groups of agents from onboard data of a mobile platform. We evaluate the benefit of interaction-aware motion planning in a realistic public setting with a total distance traveled of over 4 km. Interestingly the motion models learned from human-human interaction did not hold for robot-human interaction, due to the high attention and interest of pedestrians in testing basic braking functionality of the robot.
---
paper_title: Predicting wide receiver trajectories in American football

paper_content:

Predicting the trajectory of a wide receiver in the game of American football requires prior knowledge about the game (e.g., route trees, defensive formations) and an accurate model of how the environment will change over time (e.g., opponent reaction strategies, motion attributes of players). Our aim is to build a computational model of the wide receiver, which takes into account prior knowledge about the game and short-term predictive models of how the environment will change over time. While prior knowledge of the game is readily accessible, it is quite challenging to build predictive models of how the environment will change over time. We propose several models for predicting short-term motions of opponent players to generate dynamic input features for our wide receiver forecasting model. In particular, we model the wide receiver with a Markov Decision Process (MDP), where the reward function is a linear combination of static features (prior knowledge about the game) and dynamic features (short-term prediction of opponent players). Since the dynamic features change over time, we make recursive calls to an inference procedure over the MDP while updating the dynamic features. We validate our technique on a video dataset of American football plays. Our results show that more informed models that accurately predict the motions of the defensive players are better at forecasting wide receiver plays.
---
paper_title: Robot navigation in dense human crowds: the case for cooperation

paper_content:

We consider mobile robot navigation in dense human crowds. In particular, we explore two questions. Can we design a navigation algorithm that encourages humans to cooperate with a robot? Would such cooperation improve navigation performance? We address the first question by developing a probabilistic predictive model of cooperative collision avoidance and goal-oriented behavior by extending the interacting Gaussian processes approach to include multiple goals and stochastic movement duration. We answer the second question with an extensive quantitative study of robot navigation in dense human crowds (488 runs completed), specifically testing how cooperation models effect navigation performance. We find that the “multiple goal” interacting Gaussian processes algorithm performs comparably with human teleoperators in crowd densities near 1 person/m2, while a state of the art noncooperative planner exhibits unsafe behavior more than 3 times as often as this multiple goal extension, and more than twice as often as the basic interacting Gaussian processes. Furthermore, a reactive planner based on the widely used “dynamic window” approach fails for crowd densities above 0.55 people/m2. Based on these experimental results, and previous theoretical observations, we conclude that a cooperation model is important for safe and efficient robot navigation in dense human crowds.
---
paper_title: Driver and pedestrian awareness-based collision risk analysis

paper_content:

We present a novel approach for vehicle-pedestrian collision risk analysis that incorporates mutual situational awareness, a degree of potential motion coupling and the spatial layout of the environment. The approach uses a Dynamic Bayesian Network (DBN) for modeling the individual object paths; collision risk is subsequently computed by an intersection operation.
---
paper_title: Goal-directed pedestrian model for long-term motion prediction with application to robot motion planning

paper_content:

A probabilistic goal-directed model is proposed for pedestrian motion using navigation function and statistics of human motion gathered in the environment. In comparison with existing models, this model is both computationally inexpensive and does not fail when the optimal direction of motion in terms of this model is non-unique. We further introduce a Rapidly-Exploring Random Tree (RRT)-based path planner developed for planning in state-time space. With the help of an improved distance metric, the planner is much faster than RRT-Blossom [11] in complex maps. In an environment with 10 pedestrians, the planner and motion prediction combined can perform in near-real time.
---
paper_title: Socially-aware robot navigation: A learning approach

paper_content:

The ability to act in a socially-aware way is a key skill for robots that share a space with humans. In this paper we address the problem of socially-aware navigation among people that meets objective criteria such as travel time or path length as well as subjective criteria such as social comfort. Opposed to model-based approaches typically taken in related work, we pose the problem as an unsupervised learning problem. We learn a set of dynamic motion prototypes from observations of relative motion behavior of humans found in publicly available surveillance data sets. The learned motion prototypes are then used to compute dynamic cost maps for path planning using an any-angle A* algorithm. In the evaluation we demonstrate that the learned behaviors are better in reproducing human relative motion in both criteria than a Proxemics-based baseline method.
---
paper_title: Building Prior Knowledge: A Markov Based Pedestrian Prediction Model Using Urban Environmental Data

paper_content:

Autonomous Vehicles navigating in urban areas have a need to understand and predict future pedestrian behavior for safer navigation. This high level of situational awareness requires observing pedestrian behavior and extrapolating their positions to know future positions. While some work has been done in this field using Hidden Markov Models (HMMs), one of the few observed drawbacks of the method is the need for informed priors for learning behavior. In this work, an extension to the Growing Hidden Markov Model (GHMM) method is proposed to solve some of these drawbacks. This is achieved by building on existing work using potential cost maps and the principle of Natural Vision. As a consequence, the proposed model is able to predict pedestrian positions more precisely over a longer horizon compared to the state of the art. The method is tested over "legal" and "illegal" behavior of pedestrians, having trained the model with sparse observations and partial trajectories. The method, with no training data, is compared against a trained state of the art model. It is observed that the proposed method is robust even in new, previously unseen areas.
---
paper_title: Online trajectory prediction and planning for social robot navigation

paper_content:

This paper addresses the safe and legible navigation of mobile robots in multi-agent encounters. A novel motion model provides the basis to predict, plan and coordinate agent trajectories in intersection scenarios. The approach establishes an implicit, non-overt cooperation between the robot and humans by linking the prediction and planning of agent trajectories within a unified representation in terms of timed elastic bands. The planning process maintains multiple topological alternatives to resolve the encounter in a manner compliant with the implicit rules and objectives of human proxemics. The trajectory is obtained by optimizing the timed elastic band considering multiple conflicting objectives such as fastest path and minimal spatial separation among agents but also global proxemic aspects such as motion coherence within a group. Cooperation is achieved by coupling predicted and planned agent trajectories to eventually reach an implicit agreement of the agents on how to circumnavigate each other. The parameters of the cost functions of the underlying motion model are identified by inverse optimal control from a dataset of 73 recorded encounters with up to five humans and a total of 283 individual trajectories. Playback simulations of recorded encounters and experiments with a robot traversing a group of oncoming humans demonstrate the feasibility of the approach to resolve general proxemic encounters.
---
paper_title: Experiment-based modeling, simulation and validation of interactions between virtual walkers

paper_content:

An interaction occurs between two humans when they walk with converging trajectories. They need to adapt their motion in order to avoid and cross one another at respectful distance. This paper presents a model for solving interactions between virtual humans. The proposed model is elaborated from experimental interactions data. We first focus our study on the pair-interaction case. In a second stage, we extend our approach to the multiple interactions case. Our experimental data allow us to state the conditions for interactions to occur between walkers, as well as each one's role during interaction and the strategies walkers set to adapt their motion. The low number of parameters of the proposed model enables its automatic calibration from available experimental data. We validate our approach by comparing simulated trajectories with real ones. We also provide comparison with previous solutions. We finally discuss the ability of our model to be extended to complex situations.
---
paper_title: A probabilistic model of human motion and navigation intent for mobile robot path planning

paper_content:

In order to effectively plan paths in environments inhabited by humans, robots must accurately predict human motion. Typical approaches to human prediction simply assume a constant velocity which is not always valid. This paper proposes to determine the likely navigation intent of humans and use that to predict human motion. Navigation intent is determined by the function and structure of the environment. Manually assigned functional places are combined with automatically extracted navigation way-points to define a number of likely navigation targets within the environment. To predict human motion toward these targets, a probabilistic model of human motion is proposed which is based on motion probability grids generated from observed motion. The models of human navigation intent and motion are integrated with an autonomous mobile robot system, with a laser range sensor detecting humans moving within the environment, and a path planning system. The models of human navigation intent and motion are verified using real captured human motion data from an office environment. Examples of human motion prediction are also presented.
---
paper_title: Vehicle Trajectory Prediction by Integrating Physics- and Maneuver-Based Approaches Using Interactive Multiple Models

paper_content:

Vehicle trajectory prediction helps automated vehicles and advanced driver-assistance systems have a better understanding of traffic environment and perform tasks such as criticality assessment in advance. In this study, an integrated vehicle trajectory prediction method is proposed by combining physics- and maneuver-based approaches. These two methods were combined for the reason that the physics-based trajectory prediction method could ensure accuracy in the short term with the consideration of vehicle running dynamic parameters, and the maneuver-based prediction approach has a long-term insight into future trajectories with maneuver estimation. In this study, the interactive multiple model trajectory prediction (IMMTP) method is proposed by combining the two predicting models. The probability of each model in the interactive multiple models could recursively adjust according to the predicting variance of each model. In addition, prediction uncertainty is considered by employing unscented Kalman filters in the physics-based prediction model. To the maneuver-based method, random elements for uncertainty are introduced to the trajectory of each maneuver inferred by using the dynamic Bayesian network. The approach is applied and analyzed in the lane-changing scenario by using naturalistic driving data. Comparison results indicate that IMMTP could achieve a more accurate prediction trajectory with a long prediction horizon.
---
paper_title: SS-LSTM: A Hierarchical LSTM Model for Pedestrian Trajectory Prediction

paper_content:

Pedestrian trajectory prediction is an extremely challenging problem because of the crowdedness and clutter of the scenes. Previous deep learning LSTM-based approaches focus on the neighbourhood influence of pedestrians but ignore the scene layouts in pedestrian trajectory prediction. In this paper, a novel hierarchical LSTM-based network is proposed to consider both the influence of social neighbourhood and scene layouts. Our SS-LSTM, which stands for Social-Scene-LSTM, uses three different LSTMs to capture person, social and scene scale information. We also use a circular shape neighbourhood setting instead of the traditional rectangular shape neighbourhood in the social scale. We evaluate our proposed method against two baseline methods and a state-of-art technique on three public datasets. The results show that our method outperforms other methods and that using circular shape neighbourhood improves the prediction accuracy.
---
paper_title: Social Attention: Modeling Attention in Human Crowds

paper_content:

Robots that navigate through human crowds need to be able to plan safe, efficient, and human predictable trajectories. This is a particularly challenging problem as it requires the robot to predict future human trajectories within a crowd where everyone implicitly cooperates with each other to avoid collisions. Previous approaches to human trajectory prediction have modeled the interactions between humans as a function of proximity. However, that is not necessarily true as some people in our immediate vicinity moving in the same direction might not be as important as other people that are further away, but that might collide with us in the future. In this work, we propose Social Attention, a novel trajectory prediction model that captures the relative importance of each person when navigating in the crowd, irrespective of their proximity. We demonstrate the performance of our method against a state-of-the-art approach on two publicly available crowd datasets and analyze the trained attention model to gain a better understanding of which surrounding agents humans attend to, when navigating in a crowd.
---
paper_title: Gaussian process regression flow for analysis of motion trajectories

paper_content:

Recognition of motions and activities of objects in videos requires effective representations for analysis and matching of motion trajectories. In this paper, we introduce a new representation specifically aimed at matching motion trajectories. We model a trajectory as a continuous dense flow field from a sparse set of vector sequences using Gaussian Process Regression. Furthermore, we introduce a random sampling strategy for learning stable classes of motions from limited data. Our representation allows for incrementally predicting possible paths and detecting anomalous events from online trajectories. This representation also supports matching of complex motions with acceleration changes and pauses or stops within a trajectory. We use the proposed approach for classifying and predicting motion trajectories in traffic monitoring domains and test on several data sets. We show that our approach works well on various types of complete and incomplete trajectories from a variety of video data sets with different frame rates.
---
paper_title: Activity forecasting

paper_content:

We address the task of inferring the future actions of people from noisy visual input. We denote this task activity forecasting. To achieve accurate activity forecasting, our approach models the effect of the physical environment on the choice of human actions. This is accomplished by the use of state-of-the-art semantic scene understanding combined with ideas from optimal control theory. Our unified model also integrates several other key elements of activity analysis, namely, destination forecasting, sequence smoothing and transfer learning. As proof-of-concept, we focus on the domain of trajectory-based activity analysis from visual input. Experimental results demonstrate that our model accurately predicts distributions over future actions of individuals. We show how the same techniques can improve the results of tracking algorithms by leveraging information about likely goals and trajectories.
---
paper_title: MX-LSTM: Mixing Tracklets and Vislets to Jointly Forecast Trajectories and Head Poses

paper_content:

Recent approaches on trajectory forecasting use tracklets to predict the future positions of pedestrians exploiting Long Short Term Memory (LSTM) architectures. This paper shows that adding vislets, that is, short sequences of head pose estimations, allows to increase significantly the trajectory forecasting performance. We then propose to use vislets in a novel framework called MX-LSTM, capturing the interplay between tracklets and vislets thanks to a joint unconstrained optimization of full covariance matrices during the LSTM backpropagation. At the same time, MX-LSTM predicts the future head poses, increasing the standard capabilities of the long-term trajectory forecasting approaches. With standard head pose estimators and an attentional-based social pooling, MX-LSTM scores the new trajectory forecasting state-of-the-art in all the considered datasets (Zara01, Zara02, UCY, and TownCentre) with a dramatic margin when the pedestrians slow down, a case where most of the forecasting approaches struggle to provide an accurate solution.
---
paper_title: A Data-driven Model for Interaction-Aware Pedestrian Motion Prediction in Object Cluttered Environments

paper_content:

This paper reports on a data-driven, interaction-aware motion prediction approach for pedestrians in environments cluttered with static obstacles. When navigating in such workspaces shared with humans, robots need accurate motion predictions of the surrounding pedestrians. Human navigation behavior is mostly influenced by their surrounding pedestrians and by the static obstacles in their vicinity. In this paper we introduce a new model based on Long-Short Term Memory (LSTM) neural networks, which is able to learn human motion behavior from demonstrated data. To the best of our knowledge, this is the first approach using LSTMs, that incorporates both static obstacles and surrounding pedestrians for trajectory forecasting. As part of the model, we introduce a new way of encoding surrounding pedestrians based on a 1d-grid in polar angle space. We evaluate the benefit of interaction-aware motion prediction and the added value of incorporating static obstacles on both simulation and real-world datasets by comparing with state-of-the-art approaches. The results show, that our new approach outperforms the other approaches while being very computationally efficient and that taking into account static obstacles for motion predictions significantly improves the prediction accuracy, especially in cluttered environments.
---
paper_title: Multi-Modal Trajectory Prediction of Surrounding Vehicles with Maneuver based LSTMs

paper_content:

To safely and efficiently navigate through complex traffic scenarios, autonomous vehicles need to have the ability to predict the future motion of surrounding vehicles. Multiple interacting agents, the multi-modal nature of driver behavior, and the inherent uncertainty involved in the task make motion prediction of surrounding vehicles a challenging problem. In this paper, we present an LSTM model for interaction aware motion prediction of surrounding vehicles on freeways. Our model assigns confidence values to maneuvers being performed by vehicles and outputs a multi-modal distribution over future motion based on them. We compare our approach with the prior art for vehicle motion prediction on the publicly available NGSIM US-101 and I-80 datasets. Our results show an improvement in terms of RMS values of prediction error. We also present an ablative analysis of the components of our proposed model and analyze the predictions made by the model in complex traffic scenarios.
---
paper_title: You'll never walk alone: Modeling social behavior for multi-target tracking

paper_content:

Object tracking typically relies on a dynamic model to predict the object's location from its past trajectory. In crowded scenarios a strong dynamic model is particularly important, because more accurate predictions allow for smaller search regions, which greatly simplifies data association. Traditional dynamic models predict the location for each target solely based on its own history, without taking into account the remaining scene objects. Collisions are resolved only when they happen. Such an approach ignores important aspects of human behavior: people are driven by their future destination, take into account their environment, anticipate collisions, and adjust their trajectories at an early stage in order to avoid them. In this work, we introduce a model of dynamic social behavior, inspired by models developed for crowd simulation. The model is trained with videos recorded from birds-eye view at busy locations, and applied as a motion model for multi-people tracking from a vehicle-mounted camera. Experiments on real sequences show that accounting for social interactions and scene knowledge improves tracking performance, especially during occlusions.
---
paper_title: Behavior estimation for a complete framework for human motion prediction in crowded environments

paper_content:

In the present work, we propose and validate a complete probabilistic framework for human motion prediction in urban or social environments. Additionally, we formulate a powerful and useful tool: the human motion behavior estimator. Three different basic behaviors have been detected: Aware, Balanced and Unaware. Our approach is based on the Social Force Model (SFM) and the intentionality prediction BHMIP. The main contribution of the present work is to make use of the behavior estimator for formulating a reliable prediction framework of human trajectories under the influence of dynamic crowds, robots, and in general any moving obstacle. Accordingly, we have demonstrated the great performance of our long-term prediction algorithm, in real scenarios, comparing to other prediction methods.
---
paper_title: Prediction of moving objects in dynamic environments using Kalman filters

paper_content:

In this work, we describe a framework for predicting future positions and orientation of moving obstacles in a time-varying environment using Kalman filtering techniques. No constraint are placed on the obstacles motion. The proposed algorithm can be used in a variety of applications, one of which is robot motion planning in time varying environments. The advantage of using this model compared to reported ones in the literature is it's ability to start the prediction process from the first time step without the need to wait for few time steps before starting the prediction process. Early experimental results are very encouraging.
---
paper_title: A Predictive Collision Avoidance Model for Pedestrian Simulation

paper_content:

We present a new local method for collision avoidance that is based on collision prediction. In our model, each pedestrian predicts possible future collisions with other pedestrians and then makes an efficient move to avoid them. Experiments show that the new approach leads to considerably shorter and less curved paths, ensuring smooth avoidance behaviour and visually compelling simulations. The method reproduces emergent behaviour like lane formation that have been observed in real crowds. The technique is easy to implement and is fast, allowing the simulation in real time of crowds of thousands of pedestrians.
---
paper_title: Sequence-to-Sequence Prediction of Vehicle Trajectory via LSTM Encoder-Decoder Architecture

paper_content:

In this paper, we propose a deep learning based vehicle trajectory prediction technique which can generate the future trajectory sequence of surrounding vehicles in real time. We employ the encoder-decoder architecture which analyzes the pattern underlying in the past trajectory using the long short-term memory (LSTM) based encoder and generates the future trajectory sequence using the LSTM based decoder. This structure produces the $K$ most likely trajectory candidates over occupancy grid map by employing the beam search technique which keeps the $K$ locally best candidates from the decoder output. The experiments conducted on highway traffic scenarios show that the prediction accuracy of the proposed method is significantly higher than the conventional trajectory prediction techniques.
---
paper_title: Mobile Agent Trajectory Prediction using Bayesian Nonparametric Reachability Trees

paper_content:

This paper presents an efficient trajectory prediction algorithm that has been developed to improve the performance of future collision avoidance and detection systems. The main idea is to embed the inferred intention information of surrounding agents into their estimated reachability sets to obtain a probabilistic description of their future paths. More specifically, the proposed approach combines the recently developed RRT-Reach algorithm and mixtures of Gaussian Processes. RRT-Reach was introduced by the authors as an extension of the closed-loop rapidly-exploring random tree (CL-RRT) algorithm to compute reachable sets of moving objects in real-time. A mixture of Gaussian processes (GP) is a flexible nonparametric Bayesian model used to represent a distribution over trajectories and have been previously demonstrated by the authors in a UAV interception and tracking of ground vehicles planning scheme. The mixture is trained using typical maneuvers learned from statistical data, and RRT-Reach utilizes samples from the GP to grow probabilistically weighted feasible paths of the surrounding vehicles. The resulting approach, denoted as RR-GP, has RRTReach’s benefits of computing trajectories that are dynamically feasible by construction, therefore efficiently approximating the reachability set of surrounding vehicles following typical patterns. RRT-GP also features the GP mixture’s benefits of providing a probabilistic weighting on the feasible trajectories produced by RRTReach, allowing our system to systematically weight trajectories by their likelihood. A demonstrative example on a car-like vehicle illustrates the advantages of the RR-GP approach by comparing it to two other GP-based algorithms.
---
paper_title: Conditional transition maps: Learning motion patterns in dynamic environments

paper_content:

In this paper we introduce a method for learning motion patterns in dynamic environments. Representations of dynamic environments have recently received an increasing amount of attention in the research community. Understanding dynamic environments is seen as one of the key challenges in order to enable autonomous navigation in real-world scenarios. However, representing the temporal dimension is a challenge yet to be solved. In this paper we introduce a spatial representation, which encapsulates the statistical dynamic behavior observed in the environment. The proposed Conditional Transition Map (CTMap) is a grid-based representation that associates a probability distribution for an object exiting the cell, given its entry direction. The transition parameters are learned from a temporal signal of occupancy on cells by using a local-neighborhood cross-correlation method. In this paper, we introduce the CTMap, the learning approach and present a proof-of-concept method for estimating future paths of dynamic objects, called Conditional Probability Propagation Tree (CPPTree). The evaluation is done using a real-world dataset collected at a busy roundabout.
---
paper_title: Context-Aware Trajectory Prediction

paper_content:

Human motion and behaviour in crowded spaces is influenced by several factors, such as the dynamics of other moving agents in the scene, as well as the static elements that might be perceived as points of attraction or obstacles. In this work, we present a new model for human trajectory prediction which is able to take advantage of both human-human and human-space interactions. The future trajectory of humans, are generated by observing their past positions and interactions with the surroundings. To this end, we propose a"context-aware"recurrent neural network LSTM model, which can learn and predict human motion in crowded spaces such as a sidewalk, a museum or a shopping mall. We evaluate our model on a public pedestrian datasets, and we contribute a new challenging dataset that collects videos of humans that navigate in a (real) crowded space such as a big museum. Results show that our approach can predict human trajectories better when compared to previous state-of-the-art forecasting models.
---
paper_title: Stochastic prediction of human motion and control of robots in the service of human

paper_content:

The authors propose a control model for human cooperative robots. According to human motion which is measured by a human recognition system, future human position is predicted. Robot trajectories are changed by acceleration/deceleration/stop so as to minimize danger, which is computed by the predicted data. In this paper, a model of a stochastic process is adopted for the prediction because it can prevent inappropriate robot motion which might be caused by prediction error. A prediction plane is divided into square cells. Each cell has probability data by which cell number, direction and speed change. They are determined by the characteristics of motion of places beforehand. Computation of the stochastic state transition chain predicts the probability that a human exists in a cell at future time. Application to a laboratory room demonstrated that this method is effective for safe motion of human cooperative robots. >
---
paper_title: People tracking with human motion predictions from social forces

paper_content:

For many tasks in populated environments, robots need to keep track of current and future motion states of people. Most approaches to people tracking make weak assumptions on human motion such as constant velocity or acceleration. But even over a short period, human behavior is more complex and influenced by factors such as the intended goal, other people, objects in the environment, and social rules. This motivates the use of more sophisticated motion models for people tracking especially since humans frequently undergo lengthy occlusion events. In this paper, we consider computational models developed in the cognitive and social science communities that describe individual and collective pedestrian dynamics for tasks such as crowd behavior analysis. In particular, we integrate a model based on a social force concept into a multi-hypothesis target tracker. We show how the refined motion predictions translate into more informed probability distributions over hypotheses and finally into a more robust tracking behavior and better occlusion handling. In experiments in indoor and outdoor environments with data from a laser range finder, the social force model leads to more accurate tracking with up to two times fewer data association errors.
---
paper_title: Unfreezing the robot: Navigation in dense, interacting crowds

paper_content:

In this paper, we study the safe navigation of a mobile robot through crowds of dynamic agents with uncertain trajectories. Existing algorithms suffer from the “freezing robot” problem: once the environment surpasses a certain level of complexity, the planner decides that all forward paths are unsafe, and the robot freezes in place (or performs unnecessary maneuvers) to avoid collisions. Since a feasible path typically exists, this behavior is suboptimal. Existing approaches have focused on reducing the predictive uncertainty for individual agents by employing more informed models or heuristically limiting the predictive covariance to prevent this overcautious behavior. In this work, we demonstrate that both the individual prediction and the predictive uncertainty have little to do with the frozen robot problem. Our key insight is that dynamic agents solve the frozen robot problem by engaging in “joint collision avoidance”: They cooperatively make room to create feasible trajectories. We develop IGP, a nonparametric statistical model based on dependent output Gaussian processes that can estimate crowd interaction from data. Our model naturally captures the non-Markov nature of agent trajectories, as well as their goal-driven navigation. We then show how planning in this model can be efficiently implemented using particle based inference. Lastly, we evaluate our model on a dataset of pedestrians entering and leaving a building, first comparing the model with actual pedestrians, and find that the algorithm either outperforms human pedestrians or performs very similarly to the pedestrians. We also present an experiment where a covariance reduction method results in highly overcautious behavior, while our model performs desirably.
---
paper_title: Behavioral Priors for Detection and Tracking of Pedestrians in Video Sequences

paper_content:

In this paper we address the problem of detection and tracking of pedestrians in complex scenarios. The inclusion of prior knowledge is more and more crucial in scene analysis to guarantee flexibility and robustness, necessary to have reliability in complex scenes. We aim to combine image processing methods with behavioral models of pedestrian dynamics, calibrated on real data. We introduce Discrete Choice Models (DCM) for pedestrian behavior and we discuss their integration in a detection and tracking context. The obtained results show how it is possible to combine both methodologies to improve the performances of such systems in complex sequences.
---
paper_title: Understanding interactions between traffic participants based on learned behaviors

paper_content:

Predicting vehicles' behaviors in a traffic scene can be very challenging due to many influences. Especially interactions with other traffic participants like vehicles or pedestrians are very crucial for the future movement while they are hard to model even with expert knowledge.
---
paper_title: Motion planning based on learning models of pedestrian and driver behaviors

paper_content:

Autonomous driving has shown the capability of providing driver convenience and enhancing safety. While introducing autonomous driving into our current traffic system, one significant issue is to make the autonomous vehicle be able to react in the same way as real human drivers. In order to ensure that an autonomous vehicle of the future will perform like human drivers, this paper proposes a vehicle motion planning model, which can represent how drivers control vehicles based on the assessment of traffic environments in the real signalized intersection. The proposed motion planning model comprises functions of pedestrian intention detection, gap detection and vehicle dynamic control. The three functions are constructed based on the analysis of actual data collected from real traffic environments. Finally, this paper demonstrates the performance of the proposed method by comparing the behaviors of our model with the behaviors of real pedestrians and human drivers. The experimental results show that our proposed model can achieve 85% recognition rate for the pedestrian crossing intention. Moreover, the vehicle controlled by the proposed motion planning model and the actual human-driven vehicle are highly similar with respect to the gap acceptance in intersections.
---
paper_title: Knowledge Transfer for Scene-specific Motion Prediction

paper_content:

When given a single frame of the video, humans can not only interpret the content of the scene, but also they are able to forecast the near future. This ability is mostly driven by their rich prior knowledge about the visual world, both in terms of (i) the dynamics of moving agents, as well as (ii) the semantic of the scene. In this work we exploit the interplay between these two key elements to predict scene-specific motion patterns. First, we extract patch descriptors encoding the probability of moving to the adjacent patches, and the probability of being in that particular patch or changing behavior. Then, we introduce a Dynamic Bayesian Network which exploits this scene specific knowledge for trajectory prediction. Experimental results demonstrate that our method is able to accurately predict trajectories and transfer predictions to a novel scene characterized by similar elements.
---
paper_title: Incremental learning of human social behaviors with feature-based spatial effects

paper_content:

In the near future, robots will exist in our home, office, hospital, school and all other public places. In other words, robots have to work and share environments with human. However, there exist many invisible social rules or social protocols in human society. For example, people usually stand in a line to access certain services. It seems that there is a social force or a “spatial effect” which forces people to perform the behavior of standing in a line. Unfortunately, these spatial effects are usually invisible and immeasurable. It makes social behavior understanding to be a difficult problem. The robot that does not comprehend these spatial effects might harm people or itself. Previous methods of behavior understanding mainly only targeted certain scenarios or environments. This paper follows the framework of SBCM (Spatial Behavior Cognition Model)[3] and extends to feature-based spatial effects. By utilizing feature to represent the spatial effect, it is easier to apply known spatial effects to different environments. This paper also demonstrates that the robot is able to learn and execute some common social behaviors under the proposed method. Several simulations and experiments are conducted to verify the proposed idea in this paper.
---
paper_title: Context-Based Path Prediction for Targets with Switching Dynamics

paper_content:

Anticipating future situations from streaming sensor data is a key perception challenge for mobile robotics and automated vehicles. We address the problem of predicting the path of objects with multiple dynamic modes. The dynamics of such targets can be described by a Switching Linear Dynamical System (SLDS). However, predictions from this probabilistic model cannot anticipate when a change in dynamic mode will occur. We propose to extract various types of cues with computer vision to provide context on the target’s behavior, and incorporate these in a Dynamic Bayesian Network (DBN). The DBN extends the SLDS by conditioning the mode transition probabilities on additional context states. We describe efficient online inference in this DBN for probabilistic path prediction, accounting for uncertainty in both measurements and target behavior. Our approach is illustrated on two scenarios in the Intelligent Vehicles domain concerning pedestrians and cyclists, so-called Vulnerable Road Users (VRUs). Here, context cues include the static environment of the VRU, its dynamic environment, and its observed actions. Experiments using stereo vision data from a moving vehicle demonstrate that the proposed approach results in more accurate path prediction than SLDS at the relevant short time horizon (1 s). It slightly outperforms a computationally more demanding state-of-the-art method.
---
paper_title: A Controlled Interactive Multiple Model Filter for Combined Pedestrian Intention Recognition and Path Prediction

paper_content:

We present a novel approach combining pedestrian intention recognition and path prediction for advanced video-based driver assistance systems. The core algorithm uses an Interacting Multiple Model Filter in combination with a Latent-dynamic Conditional Random Field model. The model integrates pedestrian dynamics and situational awareness using observations from a stereo-video system for pedestrian detection and human head pose estimation. Evaluation of our method is performed on a public available dataset addressing scenarios of lateral approaching pedestrians that might cross the road, turn into the road or stop at the curbside. During experiments, we demonstrate that the proposed approach leads to better path prediction performance in terms of a smaller lateral position error compared to state-of-the-art pedestrian intention recognition and path prediction approaches. The computational costs of our approach is comparatively low and therefore can be ported easily onto a real-time system.
---
paper_title: Pedestrian Path Prediction with Recursive Bayesian Filters: A Comparative Study

paper_content:

In the context of intelligent vehicles, we perform a comparative study on recursive Bayesian filters for pedestrian path prediction at short time horizons (< 2s). We consider Extended Kalman Filters (EKF) based on single dynamical models and Interacting Multiple Models (IMM) combining several such basic models (constant velocity/acceleration/turn). These are applied to four typical pedestrian motion types (crossing, stopping, bending in, starting). Position measurements are provided by an external state-of-the-art stereo vision-based pedestrian detector. We investigate the accuracy of position estimation and path prediction, and the benefit of the IMMs vs. the simpler single dynamical models. Special care is given to the proper sensor modeling and parameter optimization. The dataset and evaluation framework are made public to facilitate benchmarking.
---
paper_title: Voronoi tracking: location estimation using sparse and noisy sensor data

paper_content:

Tracking the activity of people in indoor environments has gained considerable attention in the robotics community over the last years. Most of the existing approaches are based on sensors, which allow to accurately determining the locations of people but do not provide means to distinguish between different persons. In this paper we propose a novel approach to tracking moving objects and their identity using noisy, sparse information collected by id-sensors such as infrared and ultrasound badge systems. The key idea of our approach is to use particle filters to estimate the locations of people on the Voronoi graph of the environment. By restricting particles to a graph, we make use of the inherent structure of indoor environments. The approach has two key advantages. First, it is by far more efficient and robust than unconstrained particle filters. Second, the Voronoi graph provides a natural discretization of human motion, which allows us to apply unsupervised learning techniques to derive typical motion patterns of the people in the environment. Experiments using a robot to collect ground-truth data indicate the superior performance of Voronoi tracking. Furthermore, we demonstrate that EM-based learning of behavior patterns increases the tracking performance and provides valuable information for high-level behavior recognition.
---
paper_title: Motion prediction of moving objects based on autoregressive model

paper_content:

In this paper, we describe a framework for predicting future positions and orientation of moving obstacles in a time-varying environment using autoregressive model (ARM) with conditional maximum likelihood estimate of the model parameters. No constraints are placed on the obstacles motion. The proposed algorithm can be used in a variety of applications, one of which is robot motion planning in time varying environments.
---
paper_title: Visual Path Prediction in Complex Scenes with Crowded Moving Objects

paper_content:

This paper proposes a novel path prediction algorithm for progressing one step further than the existing works focusing on single target path prediction. In this paper, we consider moving dynamics of co-occurring objects for path prediction in a scene that includes crowded moving objects. To solve this problem, we first suggest a two-layered probabilistic model to find major movement patterns and their cooccurrence tendency. By utilizing the unsupervised learning results from the model, we present an algorithm to find the future location of any target object. Through extensive qualitative/quantitative experiments, we show that our algorithm can find a plausible future path in complex scenes with a large number of moving objects.
---
paper_title: Road Infrastructure Indicators for Trajectory Prediction

paper_content:

Safe and comfortable path planning in a dynamic urban environment is essential to an autonomous vehicle. This requires the future trajectories of all other road users in the environment of the vehicle. These trajectories are predicted through modeling the motion and behaviour of these road users. In this work we state that for efficient trajectory prediction only motion indicators are not sufficient. Therefore, we propose using a curvilinear coordinate system with curvature as road infrastructure indicators to improve motion modeling and trajectory prediction. With experiments, we show that the curvilinear coordinate system with curvature sufficiently incorporates the road structure. Furthermore, we show that a sequence-tosequence RNN model is suitable to incorporate road curvature indicators directly into the modeling and prediction.
---
paper_title: Set-Based Prediction of Pedestrians in Urban Environments Considering Formalized Traffic Rules

paper_content:

Set-based predictions can ensure the safety of planned motions, since they provide a bounded region which includes all possible future states of nondeterministic models of other traffic participants. However, while autonomous vehicles are tested in urban environments, a set-based prediction tailored to pedestrians does not exist yet. This paper addresses this problem and presents an approach for set-based predictions of pedestrians using reachability analysis. We obtain tight over-approximations of pedestrians' reachable occupancy by incorporating the dynamics of pedestrians, contextual information, and traffic rules. In addition, since pedestrians often disregard traffic rules, our constraints automatically adapt so that such behaviors are included in the prediction. Using datasets of recorded pedestrians, we validate our proposed method and demonstrate its use for evasive maneuver planning of automated vehicles.
---
paper_title: 3DOF Pedestrian Trajectory Prediction Learned from Long-Term Autonomous Mobile Robot Deployment Data

paper_content:

This paper presents a novel 3DOF pedestrian trajectory prediction approach for autonomous mobile service robots. While most previously reported methods are based on learning of 2D positions in monocular camera images, our approach uses range-finder sensors to learn and predict 3DOF pose trajectories (i.e. 2D position plus 1D rotation within the world coordinate system). Our approach, T-Pose-LSTM (Temporal 3DOF-Pose Long-Short-Term Memory), is trained using long-term data from real-world robot deployments and aims to learn context-dependent (environment- and time-specific) human activities. Our approach incorporates long-term temporal information (i.e. date and time) with short-term pose observations as input. A sequence-to-sequence LSTM encoder-decoder is trained, which encodes observations into LSTM and then decodes as predictions. For deployment, it can perform on-the-fly prediction in real-time. Instead of using manually annotated data, we rely on a robust human detection, tracking and SLAM system, providing us with examples in a global coordinate system. We validate the approach using more than 15K pedestrian trajectories recorded in a care home environment over a period of three months. The experiment shows that the proposed T-Pose-LSTM model advances the state-of-the-art 2D-based method for human trajectory prediction in long-term mobile robot deployments.
---
paper_title: Planning-based prediction for pedestrians

paper_content:

We present a novel approach for determining robot movements that efficiently accomplish the robot's tasks while not hindering the movements of people within the environment. Our approach models the goal-directed trajectories of pedestrians using maximum entropy inverse optimal control. The advantage of this modeling approach is the generality of its learned cost function to changes in the environment and to entirely different environments. We employ the predictions of this model of pedestrian trajectories in a novel incremental planner and quantitatively show the improvement in hindrance-sensitive robot trajectory planning provided by our approach.
---
paper_title: Bi-Prediction: Pedestrian Trajectory Prediction Based on Bidirectional LSTM Classification

paper_content:

Pedestrian trajectory prediction is important in various applications such as driverless vehicles, social robots, intelligent tracking systems and space planning. Existing methods focus on analysing the influence of neighbours but ignore the effect of the intended destinations of pedestrians which also plays a key role in route planning. In this paper, we propose a novel two- stage trajectory prediction method to yield multiple prediction trajectories with different probabilities towards different destination regions in the scene. Our method, which we refer to as Bi-Prediction, uses a bidirectional LSTM architecture to automatically classify trajectories into a small number of route classes before trajectory prediction. We have evaluated our method against two baseline methods and three state-of-art methods on two benchmark datasets. Our experimental results show that the extra classification stage improves the accuracy of the predicted trajectories.
---
paper_title: A mobile robot that understands pedestrian spatial behaviors

paper_content:

In human society, there are many invisible social rules or spatial effects existing in our environments. The robot that does not comprehend these spatial effects might harm people or itself. This paper presents a spatial behavior cognition model (SBCM) to describe the spatial effects existing between people and people, people and environments. By understanding the spatial effects in human-lived environments, the robot not only predicts pedestrian intentions and trajectories but also behaves socially acceptable motions. Moreover, the concept of pedestrian ego-graph (PEG) is proposed to efficiently query pedestrian-like paths for trajectory prediction. Model evaluation and experiments are shown to verify the proposed idea in this paper.
---
paper_title: Using EM to learn motion behaviors of persons with mobile robots

paper_content:

We propose a method for learning models of people's motion behaviors in indoor environments. As people move through their environments, they do not move randomly. Instead, they often engage in typical motion patterns, related to specific locations that they might be interested in approaching and specific trajectories that they might follow in doing so. Knowledge about such patterns may enable a mobile robot to develop improved people following and obstacle avoidance skills. This paper proposes an algorithm that learns collections of typical trajectories that characterize a person's motion patterns. Data, recorded by mobile robots equipped with laser-range finders, is clustered into different types of motion using the popular expectation maximization algorithm, while simultaneously learning multiple motion patterns. Experimental results, obtained using data collected in a domestic residence and in an office building, illustrate that highly predictive models of human motion patterns can be learned.
---
paper_title: Enabling Flow Awareness for Mobile Robots in Partially Observable Environments

paper_content:

Understanding the environment is a key requirement for any autonomous robot operation. There is extensive research on mapping geometric structure and perceiving objects. However, the environment is ...
---
paper_title: Real-Time Predictive Modeling and Robust Avoidance of Pedestrians with Uncertain, Changing Intentions

paper_content:

To plan safe trajectories in urban environments, autonomous vehicles must be able to quickly assess the future intentions of dynamic agents. Pedestrians are particularly challenging to model, as their motion patterns are often uncertain and/or unknown a priori. This paper presents a novel changepoint detection and clustering algorithm that, when coupled with offline unsupervised learning of a Gaussian process mixture model (DPGP), enables quick detection of changes in intent and online learning of motion patterns not seen in prior training data. The resulting long-term movement predictions demonstrate improved accuracy relative to offline learning alone, in terms of both intent and trajectory prediction. By embedding these predictions within a chance-constrained motion planner, trajectories which are probabilistically safe to pedestrian motions can be identified in real-time. Hardware experiments demonstrate that this approach can accurately predict pedestrian motion patterns from onboard sensor/perception data and facilitate robust navigation within a dynamic environment.
---
paper_title: Long-term vehicle motion prediction

paper_content:

Future driver assistance systems will have to cope with complex traffic situations, especially in the road crossing scenario. To detect potentially hazardous situations as early as possible, it is therefore desirable to know the position and motion of the ego-vehicle and vehicles around it for several seconds in advance. For this purpose, we propose in this study a long-term prediction approach based on a combined trajectory classification and particle filter framework. As a measure for the similarity between trajectories, we introduce the quaternion-based rotationally invariant longest common subsequence (QRLCS) metric. The trajectories are classified by a radial basis function (RBF) classifier with an architecture that is able to process trajectories of arbitrary non-uniform length. The particle filter framework simultaneously tracks and assesses a large number of motion hypotheses (∼102), where the class-specific probabilities estimated by the RBF classifier are used as a-priori probabilities for the hypotheses of the particle filter. The hypotheses are clustered with a mean-shift technique and are assigned a likelihood value. Motion prediction is performed based on the cluster centre with the highest likelihood. While traditional motion prediction based on curve radius and acceleration is inaccurate especially during turning manoeuvres, we show that our approach achieves a reasonable motion prediction even for long prediction intervals of 3 s for these complex motion patterns.
---
paper_title: Real-Time Certified Probabilistic Pedestrian Forecasting

paper_content:

The success of autonomous systems will depend upon their ability to safely navigate human-centric environments. This motivates the need for a real-time, probabilistic forecasting algorithm for pedestrians, cyclists, and other agents since these predictions will form a necessary step in assessing the risk of any action. This letter presents a novel approach to probabilistic forecasting for pedestrians based on weighted sums of ordinary differential equations that are learned from historical trajectory information within a fixed scene. The resulting algorithm is embarrassingly parallel and is able to work at real-time speeds using a naive Python implementation. The quality of predicted locations of agents generated by the proposed algorithm is validated on a variety of examples and is considerably higher than existing state of the art approaches over long time horizons.
---
paper_title: Natural vision based method for predicting pedestrian behaviour in urban environments

paper_content:

This paper proposes to model pedestrian behaviour in urban scenes by combining the principles of urban planning and the sociological concept of Natural Vision. This model assumes that the environment perceived by pedestrians is composed of multiple potential fields that influence their behaviour. These fields are derived from static scene elements like side-walks, cross-walks, buildings, shops entrances and dynamic obstacles like cars and buses for instance. Using this model, autonomous cars increase their level of situational awareness in the local urban space, with the ability to infer probable pedestrian paths in the scene to predict, for example, legal and illegal crossings.
---
paper_title: Improving data association by joint modeling of pedestrian trajectories and groupings

paper_content:

We consider the problem of data association in a multiperson tracking context. In semi-crowded environments, people are still discernible as individually moving entities, that undergo many interactions with other people in their direct surrounding. Finding the correct association is therefore difficult, but higher-order social factors, such as group membership, are expected to ease the problem. However, estimating group membership is a chicken-and-egg problem: knowing pedestrian trajectories, it is rather easy to find out possible groupings in the data, but in crowded scenes, it is often difficult to estimate closely interacting trajectories without further knowledge about groups. To this end, we propose a third-order graphical model that is able to jointly estimate correct trajectories and group memberships over a short time window. A set of experiments on challenging data underline the importance of joint reasoning for data association in crowded scenarios.
---
paper_title: Transferable Pedestrian Motion Prediction Models at Intersections

paper_content:

One desirable capability of autonomous cars is to accurately predict the pedestrian motion near intersections for safe and efficient trajectory planning. We are interested in developing transfer learning algorithms that can be trained on the pedestrian trajectories collected at one intersection and yet still provide accurate predictions of the trajectories at another, previously unseen intersection. We first discussed the feature selection for transferable pedestrian motion models in general. Following this discussion, we developed one transferable pedestrian motion prediction algorithm based on Inverse Reinforcement Learning (IRL) that infers pedestrian intentions and predicts future trajectories based on observed trajectory. We evaluated our algorithm on a dataset collected at two intersections, trained at one intersection and tested at the other intersection. We used the accuracy of augmented semi-nonnegative sparse coding (ASNSC), trained and tested at the same intersection as a baseline. The result shows that the proposed algorithm improves the baseline accuracy by 40% in the non-transfer task, and 16% in the transfer task.
---
paper_title: Social force model with explicit collision prediction

paper_content:

We introduce a new specification of the social force model in which pedestrians explicitly predict the place and time of the next collision in order to avoid it. This and other specifications of the social force model are calibrated, using genetic algorithms, on a set of pedestrian trajectories, obtained tracking with laser range finders the movement of pedestrians in controlled experiments, and their performance is compared. The results show that the proposed method has a better performance in describing the trajectory set.
---
paper_title: Using road topology to improve cyclist path prediction

paper_content:

We learn motion models for cyclist path prediction on real-world tracks obtained from a moving vehicle, and propose to exploit the local road topology to obtain better predictive distributions. The tracks are extracted from the Tsinghua-Daimler Cyclist Benchmark for cyclist detection, and corrected for vehicle egomotion. Tracks are then spatially aligned to local curves and crossings in the road. We study a standard approach for path prediction in the literature based on Kalman Filters, as well as a mixture of specialized filters related to specific road orientations at junctions. Our experiments demonstrate an improved prediction accuracy (up to 20% on sharp turns) of mixing specialized motion models for canonical directions, and prior knowledge on the road topology. The new track data complements the existing video, disparity and annotation data of the original benchmark, and will be made publicly available.
---
paper_title: BRVO: Predicting pedestrian trajectories using velocity-space reasoning

paper_content:

We introduce a novel, online method to predict pedestrian trajectories using agent-based velocity-space reasoning for improved human-robot interaction and collision-free navigation. Our formulation uses velocity obstacles to model the trajectory of each moving pedestrian in a robot's environment and improves the motion model by adaptively learning relevant parameters based on sensor data. The resulting motion model for each agent is computed using statistical inferencing techniques, including a combination of ensemble Kalman filters and a maximum-likelihood estimation algorithm. This allows a robot to learn individual motion parameters for every agent in the scene at interactive rates. We highlight the performance of our motion prediction method in real-world crowded scenarios, compare its performance with prior techniques, and demonstrate the improved accuracy of the predicted trajectories. We also adapt our approach for collision-free robot navigation among pedestrians based on noisy data and highlight the results in our simulator.
---
paper_title: Modelling pedestrian trajectory patterns with Gaussian processes

paper_content:

We propose a non-parametric model for pedestrian motion based on Gaussian Process regression, in which trajectory data are modelled by regressing relative motion against current position. We show how the underlying model can be learned in an unsupervised fashion, demonstrating this on two databases collected from static surveillance cameras. We furthermore exemplify the use of model for prediction, comparing the recently proposed GP-Bayesfilters with a Monte Carlo method. We illustrate the benefit of this approach for long term motion prediction where parametric models such as Kalman Filters would perform poorly.
---
paper_title: Stochastic reachable sets of interacting traffic participants

paper_content:

Knowledge about the future development of a certain road traffic situation is indispensable for safe path planning of autonomous ground vehicles or action selection of intelligent driver assistance systems. Due to a significant uncertainty about the future behavior of traffic participants, the prediction of traffic situations should be computed in a probabilistic setting. Under consideration of the dynamics of traffic participants, their future position is computed probabilistically by Markov chains that are obtained with methods known from hybrid verification. The characteristic feature of the presented approach is that all possible behaviors of traffic participants are considered, allowing to identify any dangerous future situation. The novel contribution of this work is the explicit modeling of the interaction of traffic participants, which leads to a more accurate prediction of their positions. Results are demonstrated for different traffic situations.
---
paper_title: Unsupervised robot learning to predict person motion

paper_content:

Socially interacting robots will need to understand the intentions and recognize the behaviors of people they come in contact with. In this paper we look at how a robot can learn to recognize and predict people's intended path based on its own observations of people over time. Our approach uses people tracking on the robot from either RGBD cameras or LIDAR. The tracks are separated into homogeneous motion classes using a pre-trained SVM. Then the individual classes are clustered and prototypes are extracted from each cluster. These are then used to predict a person's future motion based on matching to a partial prototype and using the rest of the prototype as the predicted motion. Results from experiments in a kitchen environment in our lab demonstrate the capabilities of the proposed method.
---
paper_title: Probabilistic vehicle trajectory prediction over occupancy grid map via recurrent neural network

paper_content:

In this paper, we propose an efficient vehicle trajectory prediction framework based on recurrent neural network. Basically, the characteristic of the vehicle's trajectory is different from that of regular moving objects since it is affected by various latent factors including road structure, traffic rules, and driver's intention. Previous state of the art approaches use sophisticated vehicle behavior model describing these factors and derive the complex trajectory prediction algorithm, which requires a system designer to conduct intensive model optimization for practical use. Our approach is data-driven and simple to use in that it learns complex behavior of the vehicles from the massive amount of trajectory data through deep neural network model. The proposed trajectory prediction method employs the recurrent neural network called long short-term memory (LSTM) to analyze the temporal behavior and predict the future coordinate of the surrounding vehicles. The proposed scheme feeds the sequence of vehicles' coordinates obtained from sensor measurements to the LSTM and produces the probabilistic information on the future location of the vehicles over occupancy grid map. The experiments conducted using the data collected from highway driving show that the proposed method can produce reasonably good estimate of future trajectory.
---
paper_title: Intent-aware long-term prediction of pedestrian motion

paper_content:

We present a method to predict long-term motion of pedestrians, modeling their behavior as jump-Markov processes with their goal a hidden variable. Assuming approximately rational behavior, and incorporating environmental constraints and biases, including time-varying ones imposed by traffic lights, we model intent as a policy in a Markov decision process framework. We infer pedestrian state using a Rao-Blackwellized filter, and intent by planning according to a stochastic policy, reflecting individual preferences in aiming at the same goal.
---
paper_title: Modeling Spatial-Temporal Dynamics of Human Movements for Predicting Future Trajectories

paper_content:

This paper presents a novel approach to modeling the dynamics of human movements with a grid-based representation. For each grid cell, we formulate the local dynamics using a variant of the left-to-right HMM, and thus explicitly model the exiting direction from the current cell. The dependency of this process on the entry direction is captured by employing the InputOutput HMM (IOHMM). On a higher level, we introduce the place where the whole trajectory originated into the IOHMM framework forming a hierarchical input structure. Therefore, we manage to capture both local spatial-temporal correlations and the long-term dependency on faraway initiating events, thus enabling the developed model to incorporate more information and to generate more informative predictions of future trajectories. The experimental results in an office corridor environment verify the capabilities of our method.
---
paper_title: Learning intentions for improved human motion prediction

paper_content:

For many tasks, robots need to operate in human populated environments. Human motion prediction therefore is gaining importance. The concept of social forces defines virtual repelling and attracting forces from and to obstacles and points of interest. These social forces can then be used to model typical human movements given an environment and a person's intention. This work shows how such models can exploit typical motion patterns summarized by growing hidden Markov models (GHMMs) that can be learned from data online and without human intervention. An extensive series of experiments shows that exploiting the intended position estimated using a GHMM within a social forces based motion model yields a significant performance gain in comparison with the standard constant velocity-based models.
---
paper_title: Multimodal Interaction-aware Motion Prediction for Autonomous Street Crossing

paper_content:

For mobile robots navigating on sidewalks, it is essential to be able to safely cross street intersections. Most existing approaches rely on the recognition of the traffic light signal to make an informed crossing decision. Although these approaches have been crucial enablers for urban navigation, the capabilities of robots employing such approaches are still limited to navigating only on streets containing signalized intersections. In this paper, we address this challenge and propose a multimodal convolutional neural network framework to predict the safety of a street intersection for crossing. Our architecture consists of two subnetworks; an interaction-aware trajectory estimation stream IA-TCNN, that predicts the future states of all observed traffic participants in the scene, and a traffic light recognition stream AtteNet. Our IA-TCNN utilizes dilated causal convolutions to model the behavior of the observable dynamic agents in the scene without explicitly assigning priorities to the interactions among them. While AtteNet utilizes Squeeze-Excitation blocks to learn a content-aware mechanism for selecting the relevant features from the data, thereby improving the noise robustness. Learned representations from the traffic light recognition stream are fused with the estimated trajectories from the motion prediction stream to learn the crossing decision. Furthermore, we extend our previously introduced Freiburg Street Crossing dataset with sequences captured at different types of intersections, demonstrating complex interactions among the traffic participants. Extensive experimental evaluations on public benchmark datasets and our proposed dataset demonstrate that our network achieves state-of-the-art performance for each of the subtasks, as well as for the crossing safety prediction.
---
paper_title: Bayesian intention inference for trajectory prediction with an unknown goal destination

paper_content:

Contextual cues can provide a rich source of information for robots that operate in the presence of other agents such as people, animals, vehicles and fellow robots. We are interested in context, in the form of the behavioural intent of an agent, for enhanced trajectory prediction. We present a Bayesian framework that estimates both the intended goal destination and future trajectory of a mobile agent moving among multiple static obstacles. Our method is based on multi-modal hypotheses of the intended goal, and is focused primarily on the long-term trajectory of the agent. We propose a computationally efficient solution and demonstrate its behaviour in a pedestrian scenario with a real-world data set. Results show the benefits of our method in comparison to traditional trajectory prediction methods and illustrate the feasibility of integration with higher-level planning algorithms.
---
paper_title: Learning to predict trajectories of cooperatively navigating agents

paper_content:

The problem of modeling the navigation behavior of multiple interacting agents arises in different areas including robotics, computer graphics, and behavioral science. In this paper, we present an approach to learn the composite navigation behavior of interacting agents from demonstrations. The decision process that ultimately leads to the observed continuous trajectories of the agents often also comprises discrete decisions, which partition the space of composite trajectories into homotopy classes. Therefore, our method uses a mixture probability distribution that consists of a discrete distribution over the homotopy classes and continuous distributions over the composite trajectories for each homotopy class. Our approach learns the model parameters of this distribution that match, in expectation, the observed behavior in terms of user-defined features. To compute the feature expectations over the highdimensional continuous distributions, we use Hamiltonian Markov chain Monte Carlo sampling. We exploit that the distributions are highly structured due to physical constraints and guide the sampling process to regions of high probability. We apply our approach to learning the behavior of pedestrians and demonstrate that it outperforms state-of-the-art methods.
---
paper_title: Who are you with and where are you going?

paper_content:

We propose an agent-based behavioral model of pedestrians to improve tracking performance in realistic scenarios. In this model, we view pedestrians as decision-making agents who consider a plethora of personal, social, and environmental factors to decide where to go next. We formulate prediction of pedestrian behavior as an energy minimization on this model. Two of our main contributions are simple, yet effective estimates of pedestrian destination and social relationships (groups). Our final contribution is to incorporate these hidden properties into an energy formulation that results in accurate behavioral prediction. We evaluate both our estimates of destination and grouping, as well as our accuracy at prediction and tracking against state of the art behavioral model and show improvements, especially in the challenging observational situation of infrequent appearance observations–something that might occur in thousands of webcams available on the Internet.
---
paper_title: Long-term Path Prediction in Urban Scenarios using Circular Distributions

paper_content:

Abstract Human ability to foresee the near future plays a key role in everyone's life to prevent potentially dangerous situations. To be able to make predictions is crucial when people have to interact with the surrounding environment. Modeling such capability can lead to the design of automated warning systems and provide moving robots with an intelligent way of interaction with changing situation. In this work we focus on a typical urban human-scene where we aim at predicting an agent's behavior using a stochastic model. In this approach, we fuse the various factors that would contribute to a human motion in different contexts. Our method uses previously observed trajectories to build point-wise circular distributions that after combination, provide a statistical smooth prediction towards the most likely areas. More specifically, a ray-launching procedure, based on a semantic segmentation, gives a coarse scene representation for collision avoidance; a nearly-constant velocity dynamic model smooths the acceleration progression and knowledge of the agent's destination may further steer the path prediction. Experimental results in structured scenes, validate the effectiveness of the method in predicting paths in comparison to actual trajectories.
---
paper_title: A Game-Theoretic Approach to Replanning-Aware Interactive Scene Prediction and Planning

paper_content:

This paper presents a novel cooperative-driving prediction and planning framework for dynamic environments based on the methods of game theory. The proposed algorithm can be used for highly automated driving on highways or as a sophisticated prediction module for advanced driver-assistance systems with no need for intervehicle communication. The main contribution of this paper is a model-based interaction-aware motion prediction of all vehicles in a scene. In contrast to other state-of-the-art approaches, the system also models the replanning capabilities of all drivers. With that, the driving strategy is able to capture complex interactions between vehicles, thus planning maneuver sequences over longer time horizons. It also enables an accurate prediction of traffic for the next immediate time step. The prediction model is supported by an interpretation of what other drivers intend to do, how they interact with traffic, and the ongoing observation. As part of the prediction loop, the proposed planning strategy incorporates the expected reactions of all traffic participants, offering cooperative and robust driving decisions. By means of experimental results under simulated highway scenarios, the validity of the proposed concept and its real-time capability is demonstrated.
---
paper_title: Predicting trajectories of vehicles using large-scale motion priors

paper_content:

We present a simple yet effective paradigm to accurately predict the future trajectories of observed vehicles in dense city environments. We equipped a large fleet of cars with cameras and performed city-scale structure-from-motion to accurately reconstruct 10M positions of their trajectories spanning over 1000h of driving.We demonstrate that this information can be used as a powerful high-fidelity prior to predict future trajectories of newly observed vehicles in the area without the need for any knowledge of road infrastructure or vehicle motion models. By relating the current position of the observed car to a large dataset of the previously exhibited motion in the area we can directly perform prediction of its future position.We evaluate our method on two large-scale data sets from San Francisco and New York City and demonstrate an order of magnitude improvement compared to a linear-motion based method. We also demonstrate that the performance naturally improves with the amount of data and ultimately yields a system that can accurately predict vehicle motion in challenging situations across extremes in traffic, time, and weather conditions.
---
paper_title: Pedestrian's Trajectory Forecast in Public Traffic with Artificial Neural Networks

paper_content:

This paper focuses on forecasting of pedestrian's short-time trajectories up to 2.5 s for traffic safety applications. We present a self-learning approach based on artificial neural network movement models and compare it to traditional constant velocity Kalman Filter prediction and extrapolation of polynomials fitted using a least-squares error. Trajectories of uninstructed pedestrians in public traffic at a real urban intersection are acquired by a wide angle stereo camera setup in combination with a 3D head tracking framework. Results using this real-world data show that the artificial neural network significantly improves forecast quality compared to other approaches especially for critical traffic scenes including velocity changes such as starting and stopping. For those velocity changes a reduction of position estimation errors of about 21% compared to the Kalman Filter and to extrapolation of polynomials is obtained. By means of a concrete pedestrian-vehicle scenario we demonstrate the benefit of the proposed approach for an advanced driver assistant system in terms of reaction time.
---
paper_title: Learning to navigate through crowded environments

paper_content:

The goal of this research is to enable mobile robots to navigate through crowded environments such as indoor shopping malls, airports, or downtown side walks. The key research question addressed in this paper is how to learn planners that generate human-like motion behavior. Our approach uses inverse reinforcement learning (IRL) to learn human-like navigation behavior based on example paths. Since robots have only limited sensing, we extend existing IRL methods to the case of partially observable environments. We demonstrate the capabilities of our approach using a realistic crowd flow simulator in which we modeled multiple scenarios in crowded environments. We show that our planner learned to guide the robot along the flow of people when the environment is crowded, and along the shortest path if no people are around.
---
paper_title: Modeling cooperative navigation in dense human crowds

paper_content:

For robots to be a part of our daily life, they need to be able to navigate among crowds not only safely but also in a socially compliant fashion. This is a challenging problem because humans tend to navigate by implicitly cooperating with one another to avoid collisions, while heading toward their respective destinations. Previous approaches have used handcrafted functions based on proximity to model human-human and human-robot interactions. However, these approaches can only model simple interactions and fail to generalize for complex crowded settings. In this paper, we develop an approach that models the joint distribution over future trajectories of all interacting agents in the crowd, through a local interaction model that we train using real human trajectory data. The interaction model infers the velocity of each agent based on the spatial orientation of other agents in his vicinity. During prediction, our approach infers the goal of the agent from its past trajectory and uses the learned model to predict its future trajectory. We demonstrate the performance of our method against a state-of-the-art approach on a public dataset and show that our model outperforms when predicting future trajectories for longer horizons.
---
paper_title: Recognition of situation classes at road intersections

paper_content:

The recognition and prediction of situations is an indispensable skill of future driver assistance systems. This study focuses on the recognition of situations involving two vehicles at intersections. For each vehicle, a set of possible future motion trajectories is estimated and rated based on a motion database for a time interval of 2–4 seconds ahead. Realistic situations are generated by a pairwise combination of these individual motion trajectories and classified according to nine categories with a polynomial classifier. In the proposed framework, situations are penalised for which the time to collision significantly exceeds the typical human reaction time. The correspondingly favoured situations are combined by a probabilistic framework, resulting in a more reliable situation recognition and collision detection than obtained based on independent motion hypotheses. The proposed method is evaluated on a real-world differential GPS data set acquired during a test drive of 10 km, including three road intersections. Our method is typically able to recognise the situation correctly about 1–2 seconds before the distance to the intersection centre becomes minimal.
---
paper_title: Estimation of Multivehicle Dynamics by Considering Contextual Information

paper_content:

Human drivers are endowed with an inborn ability to put themselves in the position of other drivers and reason about their behavior and intended actions. State-of-the-art driving-assistance systems, on the other hand, are generally limited to physical models and ad hoc safety rules. In order to drive safely amongst humans, autonomous vehicles need to develop an understanding of the situation in the form of a high-level description of the state of traffic participants. This paper presents a probabilistic model to estimate the state of vehicles by considering interactions between drivers immersed in traffic. The model is defined within a probabilistic filtering framework; estimation and prediction are carried out with statistical inference techniques. Memory requirements increase linearly with the number of vehicles, and thus, it is possible to scale the model to complex scenarios involving many participants. The approach is validated using real-world data collected by a group of interacting ground vehicles.
---
paper_title: Will the Pedestrian Cross? A Study on Pedestrian Path Prediction

paper_content:

Future vehicle systems for active pedestrian safety will not only require a high recognition performance but also an accurate analysis of the developing traffic situation. In this paper, we present a study on pedestrian path prediction and action classification at short subsecond time intervals. We consider four representative approaches: two novel approaches (based on Gaussian process dynamical models and probabilistic hierarchical trajectory matching) that use augmented features derived from dense optical flow and two approaches as baseline that use positional information only (a Kalman filter and its extension to interacting multiple models). In experiments using stereo vision data obtained from a vehicle, we investigate the accuracy of path prediction and action classification at various time horizons, the effect of various errors (image localization, vehicle egomotion estimation), and the benefit of the proposed approaches. The scenario of interest is that of a crossing pedestrian, who might stop or continue walking at the road curbside. Results indicate similar performance of the four approaches on walking motion, with near-linear dynamics. During stopping, however, the two newly proposed approaches, with nonlinear and/or higher order models and augmented motion features, achieve a more accurate position prediction of 10-50 cm at a time horizon of 0-0.77 s around the stopping event.
---
paper_title: Intention-aware online POMDP planning for autonomous driving in a crowd

paper_content:

This paper presents an intention-aware online planning approach for autonomous driving amid many pedestrians. To drive near pedestrians safely, efficiently, and smoothly, autonomous vehicles must estimate unknown pedestrian intentions and hedge against the uncertainty in intention estimates in order to choose actions that are effective and robust. A key feature of our approach is to use the partially observable Markov decision process (POMDP) for systematic, robust decision making under uncertainty. Although there are concerns about the potentially high computational complexity of POMDP planning, experiments show that our POMDP-based planner runs in near real time, at 3 Hz, on a robot golf cart in a complex, dynamic environment. This indicates that POMDP planning is improving fast in computational efficiency and becoming increasingly practical as a tool for robot planning under uncertainty. © 2015 IEEE
---
paper_title: Pedestrian Prediction by Planning Using Deep Neural Networks

paper_content:

Accurate traffic participant prediction is the prerequisite for collision avoidance of autonomous vehicles. In this work, we propose to predict pedestrians using goal-directed planning. For this, we infer a mixture density function for possible destinations. We use these destinations as the goal states of a planning stage that performs motion prediction based on common behavior patterns. The patterns are learned by a fully convolutional network operating on maps of the environment. We show that this entire system can be modeled as one monolithic neural network and trained via inverse reinforcement learning. Experimental validation on real world data shows the system's ability to predict both, destinations and trajectories accurately.
---
paper_title: Road occupancy prediction of traffic participants

paper_content:

We predict the road occupancy of traffic participants for collision avoidance systems. The occupancy sets are computed for consecutive time intervals and contain all reachable positions of traffic participants in compliance with a proposed dynamic model. Those sets make it possible to check if planned emergency maneuvers are collision-free in all possible future scenarios. However, no algorithm exists for exactly computing the occupancy when the model forbids unrealistic behavior such as leaving road boundaries or largely exceeding speed limits. For this reason, we provide methods to tightly overapproximate occupancy sets to ensure safe emergency maneuvers. We demonstrate the applicability of the approach by numerical examples, which show that the occupancy computation is not only efficient, but also tight enough to trigger emergency maneuvers almost at the last possible point in time.
---
paper_title: Motion Prediction of Traffic Actors for Autonomous Driving using Deep Convolutional Networks.

paper_content:

Recent algorithmic improvements and hardware breakthroughs resulted in a number of success stories in the field of AI impacting our daily lives. However, despite its ubiquity AI is only just starting to make advances in what may arguably have the largest impact thus far, the nascent field of autonomous driving. In this work we discuss this important topic and address one of crucial aspects of the emerging area, the problem of predicting future state of autonomous vehicle's surrounding necessary for safe and efficient operations. We introduce a deep learning-based approach that takes into account current state of traffic actors and produces rasterized representations of each actor's vicinity. The raster images are then used by deep convolutional models to infer future movement of actors while accounting for inherent uncertainty of the prediction task. Extensive experiments on real-world data strongly suggest benefits of the proposed approach. Moreover, following successful tests the system was deployed to a fleet of autonomous vehicles.
---
paper_title: Modelling Smooth Paths Using Gaussian Processes

paper_content:

A generative model based on the gaussian mixture model and gaussian processes is presented in this paper. Typical motion paths are learnt and then used for motion prediction using this model. The principal novel aspect of this approach is the modelling of paths using gaussian processes. It allows the representation of smooth trajectories and avoids discretization problems found in most existing methods. Gaussian processes not only provides a comprehensive and formal theoretical framework to work with, it also lends itself naturally to path clustering using gaussian mixture models. Learning is performed using expectation maximization where the E-Step uses variational methods to maximize its lower bound before optimization over parameters are performed in the M-Step.
---
paper_title: Pedestrian Reactive Navigation for Crowd Simulation: a Predictive Approach Abstract

paper_content:

This paper addresses the problem of virtual pedestrian autonomous navigation for crowd simulation. It describes a method for solving interactions between pedestrians and avoiding inter-collisions. Our approach is agent-based and predictive: each agent perceives surrounding agents and extrapolates their trajectory in order to react to potential collisions. We aim at obtaining realistic results, thus the proposed model is calibrated from experimental motion capture data. Our method is shown to be valid and solves major drawbacks compared to previous approaches such as oscillations due to a lack of anticipation. We first describe the mathematical representation used in our model, we then detail its implementation, and finally, its calibration and validation from real data.
---
paper_title: Map-based long term motion prediction for vehicles in traffic environments

paper_content:

Depending on driver intention and current motion state of vehicle, an infinite set of possible future trajectories exists. In this paper we present a stochastic filter which is able to select a representative set of reasonable trajectories from this solution set using additional information from a digital map. This is achieved by representing the map's traffic lanes by their corresponding centerlines. Each detected vehicle is projected on these centerlines, where the necessary information is modeled as a multivariate random variable. With this stochastic model of traffic lanes and the stochastic vector of vehicle detection we define a new stochastic residual vector, which is used both in the lane assignment and during the generation of the motion predictions. For each traffic lane, which is assigned as relevant for a detected vehicle, we generate a new motion hypothesis by using an Extended Kalman Filter. To assess the plausibility of each motion hypothesis we employ an adaptive multivariate MCUSUM algorithm. The implemented stochastic filter is tested by evaluating real data from the Ko-FAS Research Initiative. While traditional motion prediction based on previous and current motion states of a vehicle only provide plausible prediction for a short term time horizon, we show that our approach achieves a reasonable motion hypothesis for long prediction intervals especially in complex scenarios as road intersections.
---
paper_title: Pedestrian Trajectory Prediction in Extremely Crowded Scenarios

paper_content:

Pedestrian trajectory prediction under crowded circumstances is a challenging problem owing to human interaction and the complexity of the trajectory pattern. Various methods have been proposed for solving this problem, ranging from traditional Bayesian analysis to Social Force model and deep learning methods. However, most existing models heavily depend on specific scenarios because the trajectory model is constructed in absolute coordinates even though the motion trajectory as well as human interaction are in relative motion. In this study, a novel trajectory prediction model is proposed to capture the relative motion of pedestrians in extremely crowded scenarios. Trajectory sequences and human interaction are first represented with relative motion and then integrated to our model to predict pedestrians’ trajectories. The proposed model is based on Long Short Term Memory (LSTM) structure and consists of an encoder and a decoder which are trained by truncated back propagation. In addition, an anisotropic neighborhood setting is proposed instead of traditional neighborhood analysis. The proposed approach is validated using trajectory data acquired at an extremely crowded train station in Tokyo, Japan. The trajectory prediction experiments demonstrated that the proposed method outperforms existing methods and is stable for predictions of varying length even when the model is trained with a controlled short trajectory sequence.
---
paper_title: GLMP- realtime pedestrian path prediction using global and local movement patterns

paper_content:

We present a novel real-time algorithm to predict the path of pedestrians in cluttered environments. Our approach makes no assumption about pedestrian motion or crowd density, and is useful for short-term as well as long-term prediction. We interactively learn the characteristics of pedestrian motion and movement patterns from 2D trajectories using Bayesian inference. These include local movement patterns corresponding to the current and preferred velocities and global characteristics such as entry points and movement features. Our approach involves no precomputation and we demonstrate the real-time performance of our prediction algorithm on sparse and noisy trajectory data extracted from dense indoor and outdoor crowd videos. The combination of local and global movement patterns can improve the accuracy of long-term prediction by 12–18% over prior methods in high-density videos.
---
paper_title: Pedestrian Behavior Modeling From Stationary Crowds With Applications to Intelligent Surveillance

paper_content:

Pedestrian behavior modeling and analysis is important for crowd scene understanding and has various applications in video surveillance. Stationary crowd groups are a key factor influencing pedestrian walking patterns but was mostly ignored in the literature. It plays different roles for different pedestrians in a crowded scene and can change over time. In this paper, a novel model is proposed to model pedestrian behaviors by incorporating stationary crowd groups as a key component. Through inference on the interactions between stationary crowd groups and pedestrians, our model can be used to investigate pedestrian behaviors. The effectiveness of the proposed model is demonstrated through multiple applications, including walking path prediction, destination prediction, personality attribute classification, and abnormal event detection. To evaluate our model, two large pedestrian walking route datasets are built. The walking routes of around 15 000 pedestrians from two crowd surveillance videos are manually annotated. The datasets will be released to the public and benefit future research on pedestrian behavior analysis and crowd scene understanding.
---
paper_title: DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents

paper_content:

We introduce a Deep Stochastic IOC RNN Encoder-decoder framework, DESIRE, for the task of future predictions of multiple interacting agents in dynamic scenes. DESIRE effectively predicts future locations of objects in multiple scenes by 1) accounting for the multi-modal nature of the future prediction (i.e., given the same context, future may vary), 2) foreseeing the potential future outcomes and make a strategic prediction based on that, and 3) reasoning not only from the past motion history, but also from the scene context as well as the interactions among the agents. DESIRE achieves these in a single end-to-end trainable neural network model, while being computationally efficient. The model first obtains a diverse set of hypothetical future prediction samples employing a conditional variational auto-encoder, which are ranked and refined by the following RNN scoring-regression module. Samples are scored by accounting for accumulated future rewards, which enables better long-term strategic decisions similar to IOC frameworks. An RNN scene context fusion module jointly captures past motion histories, the semantic scene context and interactions among multiple agents. A feedback mechanism iterates over the ranking and refinement to further boost the prediction accuracy. We evaluate our model on two publicly available datasets: KITTI and Stanford Drone Dataset. Our experiments show that the proposed model significantly improves the prediction accuracy compared to other baseline methods.
---
paper_title: Augmented dictionary learning for motion prediction

paper_content:

Developing accurate models and efficient representations of multivariate trajectories is important for understanding the behavior patterns of mobile agents. This work presents a dictionary learning algorithm for developing a part-based trajectory representation, which combines merits of the existing Markovian-based and clustering-based approaches. In particular, this work presents the augmented semi-nonnegative sparse coding (ASNSC) algorithm for solving a constrained dictionary learning problem, and shows that the proposed method would converge to a local optimum given a convexity condition. We consider a trajectory modeling application, in which the learned dictionary atoms correspond to local motion patterns. Classical semi-nonnegative sparse coding approaches would add dictionary atoms with opposite signs to reduce the representational error, which can lead to learning noisy dictionary atoms that correspond poorly to local motion patterns. ASNSC addresses this problem and learns a concise set of intuitive motion patterns. ASNSC shows significant improvement over existing trajectory modeling methods in both prediction accuracy and computational time, as revealed by extensive numerical analysis on real datasets.
---
paper_title: Location-Velocity Attention for Pedestrian Trajectory Prediction

paper_content:

Pedestrian path forecasting is crucial in applications such as smart video surveillance. It is a challenging task because of the complex crowd movement patterns in the scenes. Most of existing state-of-the-art LSTM based prediction methods require rich context like labelled static obstacles, labelled entrance/exit regions and even the background scene. Furthermore, incorporating contextual information into trajectory prediction increases the computational overhead and decreases the generalization of the prediction models across different scenes. In this paper, we propose a joint Location-Velocity Attention LSTM based method to predict trajectories. Specifically, a module is designed to tweak the LSTM network and an attention mechanism is trained to learn to optimally combine the location and the velocity information of pedestrians in the prediction process. We have evaluated our approach against other baselines and state-of-the-art methods on several publicly available datasets. The results show that it not only outperforms other prediction methods but it also has a good generalization ability.
---
paper_title: SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints

paper_content:

This paper addresses the problem of path prediction for multiple interacting agents in a scene, which is a crucial step for many autonomous platforms such as self-driving cars and social robots. We present SoPhie; an interpretable framework based on Generative Adversarial Network (GAN), which leverages two sources of information, the path history of all the agents in a scene, and the scene context information, using images of the scene. To predict a future path for an agent, both physical and social information must be leveraged. Previous work has not been successful to jointly model physical and social interactions. Our approach blends a social attention mechanism with physical attention that helps the model to learn where to look in a large scene and extract the most salient parts of the image relevant to the path. Whereas, the social attention component aggregates information across the different agent interactions and extracts the most important trajectory information from the surrounding neighbors. SoPhie also takes advantage of GAN to generates more realistic samples and to capture the uncertain nature of the future paths by modeling its distribution. All these mechanisms enable our approach to predict socially and physically plausible paths for the agents and to achieve state-of-the-art performance on several different trajectory forecasting benchmarks.
---
paper_title: A velocity-based approach for simulating human collision avoidance

paper_content:

We present a velocity-based model for realistic collision avoidance among virtual characters. Our approach is elaborated from experimental data and is based on the simple hypothesis that an individual tries to resolve collisions long in advance by slightly adapting its motion.
---
paper_title: INFER: INtermediate representations for FuturE pRediction

paper_content:

In urban driving scenarios, forecasting future trajectories of surrounding vehicles is of paramount importance. While several approaches for the problem have been proposed, the best-performing ones tend to require extremely detailed input representations (e.g. image sequences). As a result, such methods do not generalize to datasets they have not been trained on. In this paper, we propose intermediate representations that are particularly well-suited for future prediction. As opposed to using texture (color) information from images, we condition on semantics and train an autoregressive model to accurately predict future trajectories of traffic participants (vehicles) (see Fig. above). We demonstrate that semantics provide a significant boost over techniques that operate over raw pixel intensities/disparities. Uncharacteristic of state-of-the-art approaches, our representations and models generalize across different sensing modalities (stereo imagery, LiDAR, a combination of both), and also across completely different datasets, collected across several cities, and also across countries where people drive on opposite sides of the road (left-handed vs right-handed driving). Additionally, we demonstrate an application of our approach in multi-object tracking (data association). To foster further research in transferable representations and ensure reproducibility, we release all our code and data.33More qualitative and quantitative results, along with code and data can be found at https://rebrand.ly/INFER-results
---
paper_title: Human Motion Behaviour Aware Planner (HMBAP) for path planning in dynamic human environments

paper_content:

For a robot navigating in a human inhabited dynamic environment, the knowledge of how the robot's movement can influence the trajectory of people around it can be very valuable. In this work we present a Human Motion Behaviour Aware Planner (HMBAP) which incorporates a Human Motion Behaviour Model (HMBM) in its planning stage to take advantage of this. HMBM is an obstacle avoidance model for people based on social forces which gives the robot an understanding of how people would react to its planned path. This information is useful for the robot to avoid imminent collisions with people in constricted spaces. The resulting robot behaviour is similar to how a human would move (in terms of avoidance behaviour) in a similar situation. We believe that this is a desirable feature for a robot navigating in a human inhabited environment. Our method shows good human-like navigation behaviour in situations where past methods fail to find a solution.
---
paper_title: Building a human behavior map from local observations

paper_content:

This paper presents a novel method for classifying regions from human movements in service robots' working environments. The entire space is segmented subject to the class type according to the functionality or affordance of each place which accommodates a typical human behavior. This is achieved based on a grid map in two steps. First a probabilistic model is developed to capture human movements for each grid cell by using a non-ergodic HMM. Then the learned transition probabilities corresponding to these movements are used to cluster all cells by using the K-means algorithm. The knowledge of typical human movements for each location, represented by the prototypes from K-means and summarized in a ‘behavior-based map’, enables a robot to adjust the strategy of interacting with people according to where they are located, and thus greatly enhances its capability to assist people. The performance of the proposed classification method is demonstrated by experimental results from 8 hours of data that are collected in a kitchen environment.
---
paper_title: Human Trajectory Prediction using Spatially aware Deep Attention Models

paper_content:

Trajectory Prediction of dynamic objects is a widely studied topic in the field of artificial intelligence. Thanks to a large number of applications like predicting abnormal events, navigation system for the blind, etc. there have been many approaches to attempt learning patterns of motion directly from data using a wide variety of techniques ranging from hand-crafted features to sophisticated deep learning models for unsupervised feature learning. All these approaches have been limited by problems like inefficient features in the case of hand crafted features, large error propagation across the predicted trajectory and no information of static artefacts around the dynamic moving objects. We propose an end to end deep learning model to learn the motion patterns of humans using different navigational modes directly from data using the much popular sequence to sequence model coupled with a soft attention mechanism. We also propose a novel approach to model the static artefacts in a scene and using these to predict the dynamic trajectories. The proposed method, tested on trajectories of pedestrians, consistently outperforms previously proposed state of the art approaches on a variety of large scale data sets. We also show how our architecture can be naturally extended to handle multiple modes of movement (say pedestrians, skaters, bikers and buses) simultaneously.
---
paper_title: Probabilistic Autonomous Robot Navigation in Dynamic Environments with Human Motion Prediction

paper_content:

This paper considers the problem of autonomous robot navigation in dynamic and congested environments. The predictive navigation paradigm is proposed where probabilistic planning is integrated with obstacle avoidance along with future motion prediction of humans and/or other obstacles. Predictive navigation is performed in a global manner with the use of a hierarchical Partially Observable Markov Decision Process (POMDP) that can be solved on-line at each time step and provides the actual actions the robot performs. Obstacle avoidance is performed within the predictive navigation model with a novel approach by deciding paths to the goal position that are not obstructed by other moving objects movement with the use of future motion prediction and by enabling the robot to increase or decrease its speed of movement or by performing detours. The robot is able to decide which obstacle avoidance behavior is optimal in each case within the unified navigation model employed.
---
paper_title: A Bayesian nonparametric approach to modeling motion patterns

paper_content:

The most difficult--and often most essential--aspect of many interception and tracking tasks is constructing motion models of the targets. Experts rarely can provide complete information about a target's expected motion pattern, and fitting parameters for complex motion patterns can require large amounts of training data. Specifying how to parameterize complex motion patterns is in itself a difficult task. ::: ::: In contrast, Bayesian nonparametric models of target motion are very flexible and generalize well with relatively little training data. We propose modeling target motion patterns as a mixture of Gaussian processes (GP) with a Dirichlet process (DP) prior over mixture weights. The GP provides an adaptive representation for each individual motion pattern, while the DP prior allows us to represent an unknown number of motion patterns. Both automatically adjust the complexity of the motion model based on the available data. Our approach outperforms several parametric models on a helicopter-based car-tracking task on data collected from the greater Boston area.
---
paper_title: Human-robot co-navigation using anticipatory indicators of human walking motion

paper_content:

Mobile, interactive robots that operate in human-centric environments need the capability to safely and efficiently navigate around humans. This requires the ability to sense and predict human motion trajectories and to plan around them. In this paper, we present a study that supports the existence of statistically significant biomechanical turn indicators of human walking motions. Further, we demonstrate the effectiveness of these turn indicators as features in the prediction of human motion trajectories. Human motion capture data is collected with predefined goals to train and test a prediction algorithm. Use of anticipatory features results in improved performance of the prediction algorithm. Lastly, we demonstrate the closed-loop performance of the prediction algorithm using an existing algorithm for motion planning within dynamic environments. The anticipatory indicators of human walking motion can be used with different prediction and/or planning algorithms for robotics; the chosen planning and prediction algorithm demonstrates one such implementation for human-robot co-navigation.
---
paper_title: Camera-based observation of obstacle motions to derive statistical data for mobile robot motion planning

paper_content:

Mobile robots for advanced applications have to act in environments which contain moving obstacles. Motions of obstacles (e.g. humans) usually are not precisely predictable, but neither they are completely random. Long-term observation of obstacle behavior may yield knowledge about prevailing motion patterns. This paper presents concepts and results of an experimental system. Using cameras mounted at the ceiling, the workspace is observed. The image data is processed and transformed into a compact statistical representation of motion patterns. Mobile robot motion planning benefits from such additional knowledge: trajectories are rated in terms of collision probability and expected time for reaching the goal; planning yields efficient paths which are adapted to obstacle behavior.
---
paper_title: Learning Social Etiquette: Human Trajectory Understanding In Crowded Scenes

paper_content:

Humans navigate crowded spaces such as a university campus by following common sense rules based on social etiquette. In this paper, we argue that in order to enable the design of new target tracking or trajectory forecasting methods that can take full advantage of these rules, we need to have access to better data in the first place. To that end, we contribute a new large-scale dataset that collects videos of various types of targets (not just pedestrians, but also bikers, skateboarders, cars, buses, golf carts) that navigate in a real world outdoor environment such as a university campus. Moreover, we introduce a new characterization that describes the “social sensitivity” at which two targets interact. We use this characterization to define “navigation styles” and improve both forecasting models and state-of-the-art multi-target tracking–whereby the learnt forecasting models help the data association step.
---
paper_title: Group LSTM: Group Trajectory Prediction in Crowded Scenarios

paper_content:

The analysis of crowded scenes is one of the most challenging scenarios in visual surveillance, and a variety of factors need to be taken into account, such as the structure of the environments, and the presence of mutual occlusions and obstacles. Traditional prediction methods (such as RNN, LSTM, VAE, etc.) focus on anticipating individual’s future path based on the precise motion history of a pedestrian. However, since tracking algorithms are generally not reliable in highly dense scenes, these methods are not easily applicable in real environments. Nevertheless, it is very common that people (friends, couples, family members, etc.) tend to exhibit coherent motion patterns. Motivated by this phenomenon, we propose a novel approach to predict future trajectories in crowded scenes, at the group level. First, by exploiting the motion coherency, we cluster trajectories that have similar motion trends. In this way, pedestrians within the same group can be well segmented. Then, an improved social-LSTM is adopted for future path prediction. We evaluate our approach on standard crowd benchmarks (the UCY dataset and the ETH dataset), demonstrating its efficacy and applicability.
---
paper_title: Multi-Agent Tensor Fusion for Contextual Trajectory Prediction

paper_content:

Accurate prediction of others' trajectories is essential for autonomous driving. Trajectory prediction is challenging because it requires reasoning about agents' past movements, social interactions among varying numbers and kinds of agents, constraints from the scene context, and the stochasticity of human behavior. Our approach models these interactions and constraints jointly within a novel Multi-Agent Tensor Fusion (MATF) network. Specifically, the model encodes multiple agents' past trajectories and the scene context into a Multi-Agent Tensor, then applies convolutional fusion to capture multiagent interactions while retaining the spatial structure of agents and the scene context. The model decodes recurrently to multiple agents' future trajectories, using adversarial loss to learn stochastic predictions. Experiments on both highway driving and pedestrian crowd datasets show that the model achieves state-of-the-art prediction accuracy.
---
paper_title: Pedestrian Path, Pose, and Intention Prediction Through Gaussian Process Dynamical Models and Pedestrian Activity Recognition

paper_content:

According to several reports published by worldwide organizations, thousands of pedestrians die in road accidents every year. Due to this fact, vehicular technologies have been evolving with the intent of reducing these fatalities. This evolution has not finished yet, since, for instance, the predictions of pedestrian paths could improve the current automatic emergency braking systems. For this reason, this paper proposes a method to predict future pedestrian paths, poses, and intentions up to 1 s in advance. This method is based on balanced Gaussian process dynamical models (B-GPDMs), which reduce the 3-D time-related information extracted from key points or joints placed along pedestrian bodies into low-dimensional spaces. The B-GPDM is also capable of inferring future latent positions and reconstruct their associated observations. However, learning a generic model for all kinds of pedestrian activities normally provides less accurate predictions. For this reason, the proposed method obtains multiple models of four types of activity, i.e., walking, stopping, starting, and standing, and selects the most similar model to estimate future pedestrian states. This method detects starting activities 125 ms after the gait initiation with an accuracy of 80% and recognizes stopping intentions 58.33 ms before the event with an accuracy of 70%. Concerning the path prediction, the mean error for stopping activities at a time-to-event (TTE) of 1 s is 238.01 ± 206.93 mm and, for starting actions, the mean error at a TTE of 0 s is 331.93 ± 254.73 mm.
---
paper_title: Predicting Future Agent Motions for Dynamic Environments

paper_content:

Understanding activities of people in a monitored environment is a topic of active research, motivated by applications requiring context-awareness. Inferring future agent motion is useful not only for improving tracking accuracy, but also for planning in an interactive motion task. Despite rapid advances in the area of activity forecasting, many state-of-the-art methods are still cumbersome for use in realistic robots. This is due to the requirement of having good semantic scene and map labelling, as well as assumptions made regarding possible goals and types of motion. Many emerging applications require robots with modest sensory and computational ability to robustly perform such activity forecasting in high density and dynamic environments. We address this by combining a novel multi-camera tracking method, efficient multi-resolution representations of state and a standard Inverse Reinforcement Learning (IRL) technique, to demonstrate performance that is better than the state-of-the-art in the literature. In this framework, the IRL method uses agent trajectories from a distributed tracker and estimates a reward function within a Markov Decision Process (MDP) model. This reward function can then be used to estimate the agent's motion in future novel task instances. We present empirical experiments using data gathered in our own lab and external corpora (VIRAT), based on which we find that our algorithm is not only efficiently implementable on a resource constrained platform but is also competitive in terms of accuracy with state-of-the-art alternatives (e.g., up to 20% better than the results reported in [1]).
---
paper_title: Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning

paper_content:

Finding feasible, collision-free paths for multiagent systems can be challenging, particularly in non-communicating scenarios where each agent's intent (e.g. goal) is unobservable to the others. In particular, finding time efficient paths often requires anticipating interaction with neighboring agents, the process of which can be computationally prohibitive. This work presents a decentralized multiagent collision avoidance algorithm based on a novel application of deep reinforcement learning, which effectively offloads the online computation (for predicting interaction patterns) to an offline learning procedure. Specifically, the proposed approach develops a value network that encodes the estimated time to the goal given an agent's joint configuration (positions and velocities) with its neighbors. Use of the value network not only admits efficient (i.e., real-time implementable) queries for finding a collision-free velocity vector, but also considers the uncertainty in the other agents' motion. Simulation results show more than 26 percent improvement in paths quality (i.e., time to reach the goal) when compared with optimal reciprocal collision avoidance (ORCA), a state-of-the-art collision avoidance strategy.
---
paper_title: Intentional motion on-line learning and prediction

paper_content:

Predicting motion of humans, animals and other objects which move according to internal plans is a challenging problem. Most existing approaches operate in two stages: (a) learning typical motion patterns by observing an environment and (b) predicting future motion on the basis of the learned patterns. In existing techniques, learning is performed off-line, hence, it is impossible to refine the existing knowledge on the basis of the new observations obtained during the prediction phase. We propose an approach which uses hidden Markov models (HMMs) to represent motion patterns. It is different from similar approaches because it is able to learn and predict in a concurrent fashion thanks to a novel approximate learning approach, based on the growing neural gas algorithm, which estimates both HMM parameters and structure. The found structure has the property of being a planar graph, thus enabling exact inference in linear time with respect to the number of states in the model. Our experiments demonstrate that the technique works in real-time, and is able to produce sound long-term predictions of people motion.
---
paper_title: Novel planning-based algorithms for human motion prediction

paper_content:

Human motion prediction from visual tracking is a challenging problem with a wide array of applications such as robotics, video surveillance and situation understanding. Recently, planning-based methods -which assume that people move by planning over a cost function- have emerged as one of the most promising alternatives. Nevertheless, state of the art planning based algorithms have shortcomings regarding their computational complexity and ability to predict for arbitrary time intervals. This paper addresses these shortcomings by leveraging alternative planning techniques (Fast Marching Method) and formulating efficient algorithms for goal estimation and full spatiotemporal prediction with lower complexity than comparable approaches. In preliminary experiments, the proposed method significantly outperforms the accuracy of the current state-of-the-art approach while reducing the computation time by a factor of 30 using a parallel version of our algorithm.
---
paper_title: Monte Carlo road safety reasoning

paper_content:

This paper presents a framework for reasoning about the future motion of multiple objects in a road scene. Unlike previous approaches, we do not look for known dangerous configurations of objects, but rather we reason about the future paths of all objects in the scene, and assess their danger. Monte Carlo path planning is used to generate a probability distribution for the possible future motion of every car in the scene. This framework can be used to either control the car, or to display warnings for the driver.
---
paper_title: Pedestrian path prediction using body language traits

paper_content:

Driver Assistance Systems have achieved a high level of maturity in the latest years. As an example of that, sophisticated pedestrian protection systems are already available in a number of commercial vehicles from several OEMs. However, accurate pedestrian path prediction is needed in order to go a step further in terms of safety and reliability, since it can make the difference between effective and non-effective intervention. In this paper, we consider the three-dimensional pedestrian body language in order to perform path prediction in a probabilistic framework. For this purpose, the different body parts and joints are detected using stereo vision. We propose the use of GPDM (Gaussian Process Dynamical Models) for reducing the high dimensionality of the input feature vector (composed by joints and displacement vectors) in the 3D pose space and for learning the pedestrian dynamics in a latent space. Experimental results show that accurate path prediction can be achieved at a time horizon of ≈ 0.8 s.
---
paper_title: Modelling and Predicting Rhythmic Flow Patterns in Dynamic Environments

paper_content:

We present a time-dependent probabilistic map able to model and predict flow patterns of people in indoor environments. The proposed representation models the likelihood of motion direction on a grid-based map by a set of harmonic functions, which efficiently capture long-term (minutes to weeks) variations of crowd movements over time. The evaluation, performed on data from two real environments, shows that the proposed model enables prediction of human movement patterns in the future. Potential applications include human-aware motion planning, improving the efficiency and safety of robot navigation.
---
paper_title: Goal-Directed Pedestrian Prediction

paper_content:

Recent advances in road safety have lead to a constant decline of injured traffic participants in Europe per year. Still, the number of injured pedestrians remains nearly constant. As a countermeasure, active pedestrian safety is the focus of current research, for which accurate pedestrian prediction is a prerequisite. In this scope, we propose a method for dynamics-and environment-based pedestrian prediction. We introduce the pedestrian's destination as a latent variable and thus convert the prediction problem into a planning problem. The planning is executed based on the current dynamics of the pedestrian. The distribution over the destinations is modeled using a Particle Filter. Experimental results show a significant improvement over existing approaches such as Kalman Filters.
---
paper_title: Pedestrian Behavior Prediction based on Motion Patterns for Vehicle-to-Pedestrian Collision Avoidance

paper_content:

This paper proposes a prediction method for vehicle-to-pedestrian collision avoidance, which learns and then predicts pedestrian behaviors as their motion instances are being observed. During learning, known trajectories are clustered to form motion patterns (MP), which become knowledge a priori to a multi-level prediction model that predicts long-term or short-term pedestrian behaviors. Simulation results show that it works well in a complex structured environment and the prediction is consistent with actual behaviors.
---
paper_title: Predicting actions to act predictably: Cooperative partial motion planning with maximum entropy models

paper_content:

This paper reports on a data-driven motion planning approach for interaction-aware, socially-compliant robot navigation among human agents. Autonomous mobile robots navigating in workspaces shared with human agents require motion planning techniques providing seamless integration and smooth navigation in such. Smooth integration in mixed scenarios calls for two abilities of the robot: predicting actions of others and acting predictably for them. The former requirement requests trainable models of agent behaviors in order to accurately forecast their actions in the future, taking into account their reaction on the robot's decisions. A human-like navigation style of the robot facilitates other agents—most likely not aware of the underlying planning technique applied—to predict the robot motion vice versa, resulting in smoother joint navigation. The approach presented in this paper is based on a feature-based maximum entropy model and is able to guide a robot in an unstructured, real-world environment. The model is trained to predict joint behavior of heterogeneous groups of agents from onboard data of a mobile platform. We evaluate the benefit of interaction-aware motion planning in a realistic public setting with a total distance traveled of over 4 km. Interestingly the motion models learned from human-human interaction did not hold for robot-human interaction, due to the high attention and interest of pedestrians in testing basic braking functionality of the robot.
---
paper_title: Social LSTM: Human Trajectory Prediction in Crowded Spaces

paper_content:

Pedestrians follow different trajectories to avoid obstacles and accommodate fellow pedestrians. Any autonomous vehicle navigating such a scene should be able to foresee the future positions of pedestrians and accordingly adjust its path to avoid collisions. This problem of trajectory prediction can be viewed as a sequence generation task, where we are interested in predicting the future trajectory of people based on their past positions. Following the recent success of Recurrent Neural Network (RNN) models for sequence prediction tasks, we propose an LSTM model which can learn general human movement and predict their future trajectories. This is in contrast to traditional approaches which use hand-crafted functions such as Social forces. We demonstrate the performance of our method on several public datasets. Our model outperforms state-of-the-art methods on some of these datasets. We also analyze the trajectories predicted by our model to demonstrate the motion behaviour learned by our model.
---
paper_title: Robot navigation in dense human crowds: the case for cooperation

paper_content:

We consider mobile robot navigation in dense human crowds. In particular, we explore two questions. Can we design a navigation algorithm that encourages humans to cooperate with a robot? Would such cooperation improve navigation performance? We address the first question by developing a probabilistic predictive model of cooperative collision avoidance and goal-oriented behavior by extending the interacting Gaussian processes approach to include multiple goals and stochastic movement duration. We answer the second question with an extensive quantitative study of robot navigation in dense human crowds (488 runs completed), specifically testing how cooperation models effect navigation performance. We find that the “multiple goal” interacting Gaussian processes algorithm performs comparably with human teleoperators in crowd densities near 1 person/m2, while a state of the art noncooperative planner exhibits unsafe behavior more than 3 times as often as this multiple goal extension, and more than twice as often as the basic interacting Gaussian processes. Furthermore, a reactive planner based on the widely used “dynamic window” approach fails for crowd densities above 0.55 people/m2. Based on these experimental results, and previous theoretical observations, we conclude that a cooperation model is important for safe and efficient robot navigation in dense human crowds.
---
paper_title: Driver and pedestrian awareness-based collision risk analysis

paper_content:

We present a novel approach for vehicle-pedestrian collision risk analysis that incorporates mutual situational awareness, a degree of potential motion coupling and the spatial layout of the environment. The approach uses a Dynamic Bayesian Network (DBN) for modeling the individual object paths; collision risk is subsequently computed by an intersection operation.
---
paper_title: Goal-directed pedestrian model for long-term motion prediction with application to robot motion planning

paper_content:

A probabilistic goal-directed model is proposed for pedestrian motion using navigation function and statistics of human motion gathered in the environment. In comparison with existing models, this model is both computationally inexpensive and does not fail when the optimal direction of motion in terms of this model is non-unique. We further introduce a Rapidly-Exploring Random Tree (RRT)-based path planner developed for planning in state-time space. With the help of an improved distance metric, the planner is much faster than RRT-Blossom [11] in complex maps. In an environment with 10 pedestrians, the planner and motion prediction combined can perform in near-real time.
---
paper_title: Socially-aware robot navigation: A learning approach

paper_content:

The ability to act in a socially-aware way is a key skill for robots that share a space with humans. In this paper we address the problem of socially-aware navigation among people that meets objective criteria such as travel time or path length as well as subjective criteria such as social comfort. Opposed to model-based approaches typically taken in related work, we pose the problem as an unsupervised learning problem. We learn a set of dynamic motion prototypes from observations of relative motion behavior of humans found in publicly available surveillance data sets. The learned motion prototypes are then used to compute dynamic cost maps for path planning using an any-angle A* algorithm. In the evaluation we demonstrate that the learned behaviors are better in reproducing human relative motion in both criteria than a Proxemics-based baseline method.
---
paper_title: Building Prior Knowledge: A Markov Based Pedestrian Prediction Model Using Urban Environmental Data

paper_content:

Autonomous Vehicles navigating in urban areas have a need to understand and predict future pedestrian behavior for safer navigation. This high level of situational awareness requires observing pedestrian behavior and extrapolating their positions to know future positions. While some work has been done in this field using Hidden Markov Models (HMMs), one of the few observed drawbacks of the method is the need for informed priors for learning behavior. In this work, an extension to the Growing Hidden Markov Model (GHMM) method is proposed to solve some of these drawbacks. This is achieved by building on existing work using potential cost maps and the principle of Natural Vision. As a consequence, the proposed model is able to predict pedestrian positions more precisely over a longer horizon compared to the state of the art. The method is tested over "legal" and "illegal" behavior of pedestrians, having trained the model with sparse observations and partial trajectories. The method, with no training data, is compared against a trained state of the art model. It is observed that the proposed method is robust even in new, previously unseen areas.
---
paper_title: Online trajectory prediction and planning for social robot navigation

paper_content:

This paper addresses the safe and legible navigation of mobile robots in multi-agent encounters. A novel motion model provides the basis to predict, plan and coordinate agent trajectories in intersection scenarios. The approach establishes an implicit, non-overt cooperation between the robot and humans by linking the prediction and planning of agent trajectories within a unified representation in terms of timed elastic bands. The planning process maintains multiple topological alternatives to resolve the encounter in a manner compliant with the implicit rules and objectives of human proxemics. The trajectory is obtained by optimizing the timed elastic band considering multiple conflicting objectives such as fastest path and minimal spatial separation among agents but also global proxemic aspects such as motion coherence within a group. Cooperation is achieved by coupling predicted and planned agent trajectories to eventually reach an implicit agreement of the agents on how to circumnavigate each other. The parameters of the cost functions of the underlying motion model are identified by inverse optimal control from a dataset of 73 recorded encounters with up to five humans and a total of 283 individual trajectories. Playback simulations of recorded encounters and experiments with a robot traversing a group of oncoming humans demonstrate the feasibility of the approach to resolve general proxemic encounters.
---
paper_title: Experiment-based modeling, simulation and validation of interactions between virtual walkers

paper_content:

An interaction occurs between two humans when they walk with converging trajectories. They need to adapt their motion in order to avoid and cross one another at respectful distance. This paper presents a model for solving interactions between virtual humans. The proposed model is elaborated from experimental interactions data. We first focus our study on the pair-interaction case. In a second stage, we extend our approach to the multiple interactions case. Our experimental data allow us to state the conditions for interactions to occur between walkers, as well as each one's role during interaction and the strategies walkers set to adapt their motion. The low number of parameters of the proposed model enables its automatic calibration from available experimental data. We validate our approach by comparing simulated trajectories with real ones. We also provide comparison with previous solutions. We finally discuss the ability of our model to be extended to complex situations.
---
paper_title: A probabilistic model of human motion and navigation intent for mobile robot path planning

paper_content:

In order to effectively plan paths in environments inhabited by humans, robots must accurately predict human motion. Typical approaches to human prediction simply assume a constant velocity which is not always valid. This paper proposes to determine the likely navigation intent of humans and use that to predict human motion. Navigation intent is determined by the function and structure of the environment. Manually assigned functional places are combined with automatically extracted navigation way-points to define a number of likely navigation targets within the environment. To predict human motion toward these targets, a probabilistic model of human motion is proposed which is based on motion probability grids generated from observed motion. The models of human navigation intent and motion are integrated with an autonomous mobile robot system, with a laser range sensor detecting humans moving within the environment, and a path planning system. The models of human navigation intent and motion are verified using real captured human motion data from an office environment. Examples of human motion prediction are also presented.
---
paper_title: Vehicle Trajectory Prediction by Integrating Physics- and Maneuver-Based Approaches Using Interactive Multiple Models

paper_content:

Vehicle trajectory prediction helps automated vehicles and advanced driver-assistance systems have a better understanding of traffic environment and perform tasks such as criticality assessment in advance. In this study, an integrated vehicle trajectory prediction method is proposed by combining physics- and maneuver-based approaches. These two methods were combined for the reason that the physics-based trajectory prediction method could ensure accuracy in the short term with the consideration of vehicle running dynamic parameters, and the maneuver-based prediction approach has a long-term insight into future trajectories with maneuver estimation. In this study, the interactive multiple model trajectory prediction (IMMTP) method is proposed by combining the two predicting models. The probability of each model in the interactive multiple models could recursively adjust according to the predicting variance of each model. In addition, prediction uncertainty is considered by employing unscented Kalman filters in the physics-based prediction model. To the maneuver-based method, random elements for uncertainty are introduced to the trajectory of each maneuver inferred by using the dynamic Bayesian network. The approach is applied and analyzed in the lane-changing scenario by using naturalistic driving data. Comparison results indicate that IMMTP could achieve a more accurate prediction trajectory with a long prediction horizon.
---
paper_title: SS-LSTM: A Hierarchical LSTM Model for Pedestrian Trajectory Prediction

paper_content:

Pedestrian trajectory prediction is an extremely challenging problem because of the crowdedness and clutter of the scenes. Previous deep learning LSTM-based approaches focus on the neighbourhood influence of pedestrians but ignore the scene layouts in pedestrian trajectory prediction. In this paper, a novel hierarchical LSTM-based network is proposed to consider both the influence of social neighbourhood and scene layouts. Our SS-LSTM, which stands for Social-Scene-LSTM, uses three different LSTMs to capture person, social and scene scale information. We also use a circular shape neighbourhood setting instead of the traditional rectangular shape neighbourhood in the social scale. We evaluate our proposed method against two baseline methods and a state-of-art technique on three public datasets. The results show that our method outperforms other methods and that using circular shape neighbourhood improves the prediction accuracy.
---
paper_title: Social Attention: Modeling Attention in Human Crowds

paper_content:

Robots that navigate through human crowds need to be able to plan safe, efficient, and human predictable trajectories. This is a particularly challenging problem as it requires the robot to predict future human trajectories within a crowd where everyone implicitly cooperates with each other to avoid collisions. Previous approaches to human trajectory prediction have modeled the interactions between humans as a function of proximity. However, that is not necessarily true as some people in our immediate vicinity moving in the same direction might not be as important as other people that are further away, but that might collide with us in the future. In this work, we propose Social Attention, a novel trajectory prediction model that captures the relative importance of each person when navigating in the crowd, irrespective of their proximity. We demonstrate the performance of our method against a state-of-the-art approach on two publicly available crowd datasets and analyze the trained attention model to gain a better understanding of which surrounding agents humans attend to, when navigating in a crowd.
---
paper_title: Gaussian process regression flow for analysis of motion trajectories

paper_content:

Recognition of motions and activities of objects in videos requires effective representations for analysis and matching of motion trajectories. In this paper, we introduce a new representation specifically aimed at matching motion trajectories. We model a trajectory as a continuous dense flow field from a sparse set of vector sequences using Gaussian Process Regression. Furthermore, we introduce a random sampling strategy for learning stable classes of motions from limited data. Our representation allows for incrementally predicting possible paths and detecting anomalous events from online trajectories. This representation also supports matching of complex motions with acceleration changes and pauses or stops within a trajectory. We use the proposed approach for classifying and predicting motion trajectories in traffic monitoring domains and test on several data sets. We show that our approach works well on various types of complete and incomplete trajectories from a variety of video data sets with different frame rates.
---
paper_title: Activity forecasting

paper_content:

We address the task of inferring the future actions of people from noisy visual input. We denote this task activity forecasting. To achieve accurate activity forecasting, our approach models the effect of the physical environment on the choice of human actions. This is accomplished by the use of state-of-the-art semantic scene understanding combined with ideas from optimal control theory. Our unified model also integrates several other key elements of activity analysis, namely, destination forecasting, sequence smoothing and transfer learning. As proof-of-concept, we focus on the domain of trajectory-based activity analysis from visual input. Experimental results demonstrate that our model accurately predicts distributions over future actions of individuals. We show how the same techniques can improve the results of tracking algorithms by leveraging information about likely goals and trajectories.
---
paper_title: MX-LSTM: Mixing Tracklets and Vislets to Jointly Forecast Trajectories and Head Poses

paper_content:

Recent approaches on trajectory forecasting use tracklets to predict the future positions of pedestrians exploiting Long Short Term Memory (LSTM) architectures. This paper shows that adding vislets, that is, short sequences of head pose estimations, allows to increase significantly the trajectory forecasting performance. We then propose to use vislets in a novel framework called MX-LSTM, capturing the interplay between tracklets and vislets thanks to a joint unconstrained optimization of full covariance matrices during the LSTM backpropagation. At the same time, MX-LSTM predicts the future head poses, increasing the standard capabilities of the long-term trajectory forecasting approaches. With standard head pose estimators and an attentional-based social pooling, MX-LSTM scores the new trajectory forecasting state-of-the-art in all the considered datasets (Zara01, Zara02, UCY, and TownCentre) with a dramatic margin when the pedestrians slow down, a case where most of the forecasting approaches struggle to provide an accurate solution.
---
paper_title: A Data-driven Model for Interaction-Aware Pedestrian Motion Prediction in Object Cluttered Environments

paper_content:

This paper reports on a data-driven, interaction-aware motion prediction approach for pedestrians in environments cluttered with static obstacles. When navigating in such workspaces shared with humans, robots need accurate motion predictions of the surrounding pedestrians. Human navigation behavior is mostly influenced by their surrounding pedestrians and by the static obstacles in their vicinity. In this paper we introduce a new model based on Long-Short Term Memory (LSTM) neural networks, which is able to learn human motion behavior from demonstrated data. To the best of our knowledge, this is the first approach using LSTMs, that incorporates both static obstacles and surrounding pedestrians for trajectory forecasting. As part of the model, we introduce a new way of encoding surrounding pedestrians based on a 1d-grid in polar angle space. We evaluate the benefit of interaction-aware motion prediction and the added value of incorporating static obstacles on both simulation and real-world datasets by comparing with state-of-the-art approaches. The results show, that our new approach outperforms the other approaches while being very computationally efficient and that taking into account static obstacles for motion predictions significantly improves the prediction accuracy, especially in cluttered environments.
---
paper_title: Multi-Modal Trajectory Prediction of Surrounding Vehicles with Maneuver based LSTMs

paper_content:

To safely and efficiently navigate through complex traffic scenarios, autonomous vehicles need to have the ability to predict the future motion of surrounding vehicles. Multiple interacting agents, the multi-modal nature of driver behavior, and the inherent uncertainty involved in the task make motion prediction of surrounding vehicles a challenging problem. In this paper, we present an LSTM model for interaction aware motion prediction of surrounding vehicles on freeways. Our model assigns confidence values to maneuvers being performed by vehicles and outputs a multi-modal distribution over future motion based on them. We compare our approach with the prior art for vehicle motion prediction on the publicly available NGSIM US-101 and I-80 datasets. Our results show an improvement in terms of RMS values of prediction error. We also present an ablative analysis of the components of our proposed model and analyze the predictions made by the model in complex traffic scenarios.
---
paper_title: You'll never walk alone: Modeling social behavior for multi-target tracking

paper_content:

Object tracking typically relies on a dynamic model to predict the object's location from its past trajectory. In crowded scenarios a strong dynamic model is particularly important, because more accurate predictions allow for smaller search regions, which greatly simplifies data association. Traditional dynamic models predict the location for each target solely based on its own history, without taking into account the remaining scene objects. Collisions are resolved only when they happen. Such an approach ignores important aspects of human behavior: people are driven by their future destination, take into account their environment, anticipate collisions, and adjust their trajectories at an early stage in order to avoid them. In this work, we introduce a model of dynamic social behavior, inspired by models developed for crowd simulation. The model is trained with videos recorded from birds-eye view at busy locations, and applied as a motion model for multi-people tracking from a vehicle-mounted camera. Experiments on real sequences show that accounting for social interactions and scene knowledge improves tracking performance, especially during occlusions.
---
paper_title: Inferring "Dark Matter" and "Dark Energy" from Videos

paper_content:

This paper presents an approach to localizing functional objects in surveillance videos without domain knowledge about semantic object classes that may appear in the scene. Functional objects do not have discriminative appearance and shape, but they affect behavior of people in the scene. For example, they "attract" people to approach them for satisfying certain needs (e.g., vending machines could quench thirst), or "repel" people to avoid them (e.g., grass lawns). Therefore, functional objects can be viewed as "dark matter", emanating "dark energy" that affects people's trajectories in the video. To detect "dark matter" and infer their "dark energy" field, we extend the Lagrangian mechanics. People are treated as particle-agents with latent intents to approach "dark matter" and thus satisfy their needs, where their motions are subject to a composite "dark energy" field of all functional objects in the scene. We make the assumption that people take globally optimal paths toward the intended "dark matter" while avoiding latent obstacles. A Bayesian framework is used to probabilistically model: people's trajectories and intents, constraint map of the scene, and locations of functional objects. A data-driven Markov Chain Monte Carlo (MCMC) process is used for inference. Our evaluation on videos of public squares and courtyards demonstrates our effectiveness in localizing functional objects and predicting people's trajectories in unobserved parts of the video footage.
---
paper_title: Behavior estimation for a complete framework for human motion prediction in crowded environments

paper_content:

In the present work, we propose and validate a complete probabilistic framework for human motion prediction in urban or social environments. Additionally, we formulate a powerful and useful tool: the human motion behavior estimator. Three different basic behaviors have been detected: Aware, Balanced and Unaware. Our approach is based on the Social Force Model (SFM) and the intentionality prediction BHMIP. The main contribution of the present work is to make use of the behavior estimator for formulating a reliable prediction framework of human trajectories under the influence of dynamic crowds, robots, and in general any moving obstacle. Accordingly, we have demonstrated the great performance of our long-term prediction algorithm, in real scenarios, comparing to other prediction methods.
---
paper_title: Trajectory clustering: a partition-and-group framework

paper_content:

Existing trajectory clustering algorithms group similar trajectories as a whole, thus discovering common trajectories. Our key observation is that clustering trajectories as a whole could miss common sub-trajectories. Discovering common sub-trajectories is very useful in many applications, especially if we have regions of special interest for analysis. In this paper, we propose a new partition-and-group framework for clustering trajectories, which partitions a trajectory into a set of line segments, and then, groups similar line segments together into a cluster. The primary advantage of this framework is to discover common sub-trajectories from a trajectory database. Based on this partition-and-group framework, we develop a trajectory clustering algorithm TRACLUS. Our algorithm consists of two phases: partitioning and grouping. For the first phase, we present a formal trajectory partitioning algorithm using the minimum description length(MDL) principle. For the second phase, we present a density-based line-segment clustering algorithm. Experimental results demonstrate that TRACLUS correctly discovers common sub-trajectories from real trajectory data.
---
paper_title: Long-term vehicle motion prediction

paper_content:

Future driver assistance systems will have to cope with complex traffic situations, especially in the road crossing scenario. To detect potentially hazardous situations as early as possible, it is therefore desirable to know the position and motion of the ego-vehicle and vehicles around it for several seconds in advance. For this purpose, we propose in this study a long-term prediction approach based on a combined trajectory classification and particle filter framework. As a measure for the similarity between trajectories, we introduce the quaternion-based rotationally invariant longest common subsequence (QRLCS) metric. The trajectories are classified by a radial basis function (RBF) classifier with an architecture that is able to process trajectories of arbitrary non-uniform length. The particle filter framework simultaneously tracks and assesses a large number of motion hypotheses (∼102), where the class-specific probabilities estimated by the RBF classifier are used as a-priori probabilities for the hypotheses of the particle filter. The hypotheses are clustered with a mean-shift technique and are assigned a likelihood value. Motion prediction is performed based on the cluster centre with the highest likelihood. While traditional motion prediction based on curve radius and acceleration is inaccurate especially during turning manoeuvres, we show that our approach achieves a reasonable motion prediction even for long prediction intervals of 3 s for these complex motion patterns.
---
paper_title: Using Dynamic Time Warping to Find Patterns in Time Series

paper_content:

Knowledge discovery in databases presents many interesting challenges within the context of providing computer tools for exploring large data archives. Electronic data repositories are growing quickly and contain data from commercial, scientific, and other domains. Much of this data is inherently temporal, such as stock prices or NASA telemetry data. Detecting patterns in such data streams or time series is an important knowledge discovery task. This paper describes some preliminary experiments with a dynamic programming approach to the problem. The pattern detection algorithm is based on the dynamic time warping technique used in the speech recognition field.
---
paper_title: Bayesian intention inference for trajectory prediction with an unknown goal destination

paper_content:

Contextual cues can provide a rich source of information for robots that operate in the presence of other agents such as people, animals, vehicles and fellow robots. We are interested in context, in the form of the behavioural intent of an agent, for enhanced trajectory prediction. We present a Bayesian framework that estimates both the intended goal destination and future trajectory of a mobile agent moving among multiple static obstacles. Our method is based on multi-modal hypotheses of the intended goal, and is focused primarily on the long-term trajectory of the agent. We propose a computationally efficient solution and demonstrate its behaviour in a pedestrian scenario with a real-world data set. Results show the benefits of our method in comparison to traditional trajectory prediction methods and illustrate the feasibility of integration with higher-level planning algorithms.
---
paper_title: Context-Aware Pedestrian Motion Prediction In Urban Intersections

paper_content:

This paper presents a novel context-based approach for pedestrian motion prediction in crowded, urban intersections, with the additional flexibility of prediction in similar, but new, environments. Previously, Chen et. al. combined Markovian-based and clustering-based approaches to learn motion primitives in a grid-based world and subsequently predict pedestrian trajectories by modeling the transition between learned primitives as a Gaussian Process (GP). This work extends that prior approach by incorporating semantic features from the environment (relative distance to curbside and status of pedestrian traffic lights) in the GP formulation for more accurate predictions of pedestrian trajectories over the same timescale. We evaluate the new approach on real-world data collected using one of the vehicles in the MIT Mobility On Demand fleet. The results show 12.5% improvement in prediction accuracy and a 2.65 times reduction in Area Under the Curve (AUC), which is used as a metric to quantify the span of predicted set of trajectories, such that a lower AUC corresponds to a higher level of confidence in the future direction of pedestrian motion.
---
paper_title: Extraction and clustering of motion trajectories in video

paper_content:

A system that tracks moving objects in a video dataset so as to extract a representation of the objects' 3D trajectories is described. The system then finds hierarchical clusters of similar trajectories in the video dataset. Objects' motion trajectories are extracted via an EKF formulation that provides each object's 3D trajectory up to a constant factor. To increase accuracy when occlusions occur, multiple tracking hypotheses are followed. For trajectory-based clustering and retrieval, a modified version of edit distance, called longest common subsequence is employed. Similarities are computed between projections of trajectories on coordinate axes. Trajectories are grouped based, using an agglomerative clustering algorithm. To check the validity of the approach, experiments using real data were performed.
---
paper_title: GLMP- realtime pedestrian path prediction using global and local movement patterns

paper_content:

We present a novel real-time algorithm to predict the path of pedestrians in cluttered environments. Our approach makes no assumption about pedestrian motion or crowd density, and is useful for short-term as well as long-term prediction. We interactively learn the characteristics of pedestrian motion and movement patterns from 2D trajectories using Bayesian inference. These include local movement patterns corresponding to the current and preferred velocities and global characteristics such as entry points and movement features. Our approach involves no precomputation and we demonstrate the real-time performance of our prediction algorithm on sparse and noisy trajectory data extracted from dense indoor and outdoor crowd videos. The combination of local and global movement patterns can improve the accuracy of long-term prediction by 12–18% over prior methods in high-density videos.
---
paper_title: Human Trajectory Prediction using Spatially aware Deep Attention Models

paper_content:

Trajectory Prediction of dynamic objects is a widely studied topic in the field of artificial intelligence. Thanks to a large number of applications like predicting abnormal events, navigation system for the blind, etc. there have been many approaches to attempt learning patterns of motion directly from data using a wide variety of techniques ranging from hand-crafted features to sophisticated deep learning models for unsupervised feature learning. All these approaches have been limited by problems like inefficient features in the case of hand crafted features, large error propagation across the predicted trajectory and no information of static artefacts around the dynamic moving objects. We propose an end to end deep learning model to learn the motion patterns of humans using different navigational modes directly from data using the much popular sequence to sequence model coupled with a soft attention mechanism. We also propose a novel approach to model the static artefacts in a scene and using these to predict the dynamic trajectories. The proposed method, tested on trajectories of pedestrians, consistently outperforms previously proposed state of the art approaches on a variety of large scale data sets. We also show how our architecture can be naturally extended to handle multiple modes of movement (say pedestrians, skaters, bikers and buses) simultaneously.
---
paper_title: Novel planning-based algorithms for human motion prediction

paper_content:

Human motion prediction from visual tracking is a challenging problem with a wide array of applications such as robotics, video surveillance and situation understanding. Recently, planning-based methods -which assume that people move by planning over a cost function- have emerged as one of the most promising alternatives. Nevertheless, state of the art planning based algorithms have shortcomings regarding their computational complexity and ability to predict for arbitrary time intervals. This paper addresses these shortcomings by leveraging alternative planning techniques (Fast Marching Method) and formulating efficient algorithms for goal estimation and full spatiotemporal prediction with lower complexity than comparable approaches. In preliminary experiments, the proposed method significantly outperforms the accuracy of the current state-of-the-art approach while reducing the computation time by a factor of 30 using a parallel version of our algorithm.
---
paper_title: How good is my prediction? Finding a similarity measure for trajectory prediction evaluation

paper_content:

The reliable prediction of traffic participants' trajectories is an important challenge for automated driving. Prediction methods that try to deal with this challenge need similarity measures for trajectories in order to evaluate the quality of their prediction. Currently there exists no commonly accepted similarity measure suitable for this task. In this paper we review common trajectory similarity measures and analyze them with regard to prediction evaluation. Further we introduce a new approach for synthesizing a hybrid measure that combines a set of similarity measures and provide a heuristic to determine the parameters for this approach.
---
paper_title: You'll never walk alone: Modeling social behavior for multi-target tracking

paper_content:

Object tracking typically relies on a dynamic model to predict the object's location from its past trajectory. In crowded scenarios a strong dynamic model is particularly important, because more accurate predictions allow for smaller search regions, which greatly simplifies data association. Traditional dynamic models predict the location for each target solely based on its own history, without taking into account the remaining scene objects. Collisions are resolved only when they happen. Such an approach ignores important aspects of human behavior: people are driven by their future destination, take into account their environment, anticipate collisions, and adjust their trajectories at an early stage in order to avoid them. In this work, we introduce a model of dynamic social behavior, inspired by models developed for crowd simulation. The model is trained with videos recorded from birds-eye view at busy locations, and applied as a motion model for multi-people tracking from a vehicle-mounted camera. Experiments on real sequences show that accounting for social interactions and scene knowledge improves tracking performance, especially during occlusions.
---
paper_title: Context-Based Path Prediction for Targets with Switching Dynamics

paper_content:

Anticipating future situations from streaming sensor data is a key perception challenge for mobile robotics and automated vehicles. We address the problem of predicting the path of objects with multiple dynamic modes. The dynamics of such targets can be described by a Switching Linear Dynamical System (SLDS). However, predictions from this probabilistic model cannot anticipate when a change in dynamic mode will occur. We propose to extract various types of cues with computer vision to provide context on the target’s behavior, and incorporate these in a Dynamic Bayesian Network (DBN). The DBN extends the SLDS by conditioning the mode transition probabilities on additional context states. We describe efficient online inference in this DBN for probabilistic path prediction, accounting for uncertainty in both measurements and target behavior. Our approach is illustrated on two scenarios in the Intelligent Vehicles domain concerning pedestrians and cyclists, so-called Vulnerable Road Users (VRUs). Here, context cues include the static environment of the VRU, its dynamic environment, and its observed actions. Experiments using stereo vision data from a moving vehicle demonstrate that the proposed approach results in more accurate path prediction than SLDS at the relevant short time horizon (1 s). It slightly outperforms a computationally more demanding state-of-the-art method.
---
paper_title: Forecasting Interactive Dynamics of Pedestrians with Fictitious Play

paper_content:

We develop predictive models of pedestrian dynamics by encoding the coupled nature of multi-pedestrian interaction using game theory, and deep learning-based visual analysis to estimate person-specific behavior parameters. Building predictive models for multi-pedestrian interactions however, is very challenging due to two reasons: (1) the dynamics of interaction are complex interdependent processes, where the predicted behavior of one pedestrian can affect the actions taken by others and (2) dynamics are variable depending on an individuals physical characteristics (e.g., an older person may walk slowly while the younger person may walk faster). To address these challenges, we (1) utilize concepts from game theory to model the interdependent decision making process of multiple pedestrians and (2) use visual classifiers to learn a mapping from pedestrian appearance to behavior parameters. We evaluate our proposed model on several public multiple pedestrian interaction video datasets. Results show that our strategic planning model explains human interactions 25% better when compared to state-of-the-art methods.
---
paper_title: Long-term Path Prediction in Urban Scenarios using Circular Distributions

paper_content:

Abstract Human ability to foresee the near future plays a key role in everyone's life to prevent potentially dangerous situations. To be able to make predictions is crucial when people have to interact with the surrounding environment. Modeling such capability can lead to the design of automated warning systems and provide moving robots with an intelligent way of interaction with changing situation. In this work we focus on a typical urban human-scene where we aim at predicting an agent's behavior using a stochastic model. In this approach, we fuse the various factors that would contribute to a human motion in different contexts. Our method uses previously observed trajectories to build point-wise circular distributions that after combination, provide a statistical smooth prediction towards the most likely areas. More specifically, a ray-launching procedure, based on a semantic segmentation, gives a coarse scene representation for collision avoidance; a nearly-constant velocity dynamic model smooths the acceleration progression and knowledge of the agent's destination may further steer the path prediction. Experimental results in structured scenes, validate the effectiveness of the method in predicting paths in comparison to actual trajectories.
---
paper_title: Predicting trajectories of vehicles using large-scale motion priors

paper_content:

We present a simple yet effective paradigm to accurately predict the future trajectories of observed vehicles in dense city environments. We equipped a large fleet of cars with cameras and performed city-scale structure-from-motion to accurately reconstruct 10M positions of their trajectories spanning over 1000h of driving.We demonstrate that this information can be used as a powerful high-fidelity prior to predict future trajectories of newly observed vehicles in the area without the need for any knowledge of road infrastructure or vehicle motion models. By relating the current position of the observed car to a large dataset of the previously exhibited motion in the area we can directly perform prediction of its future position.We evaluate our method on two large-scale data sets from San Francisco and New York City and demonstrate an order of magnitude improvement compared to a linear-motion based method. We also demonstrate that the performance naturally improves with the amount of data and ultimately yields a system that can accurately predict vehicle motion in challenging situations across extremes in traffic, time, and weather conditions.
---
paper_title: Predicting Future Agent Motions for Dynamic Environments

paper_content:

Understanding activities of people in a monitored environment is a topic of active research, motivated by applications requiring context-awareness. Inferring future agent motion is useful not only for improving tracking accuracy, but also for planning in an interactive motion task. Despite rapid advances in the area of activity forecasting, many state-of-the-art methods are still cumbersome for use in realistic robots. This is due to the requirement of having good semantic scene and map labelling, as well as assumptions made regarding possible goals and types of motion. Many emerging applications require robots with modest sensory and computational ability to robustly perform such activity forecasting in high density and dynamic environments. We address this by combining a novel multi-camera tracking method, efficient multi-resolution representations of state and a standard Inverse Reinforcement Learning (IRL) technique, to demonstrate performance that is better than the state-of-the-art in the literature. In this framework, the IRL method uses agent trajectories from a distributed tracker and estimates a reward function within a Markov Decision Process (MDP) model. This reward function can then be used to estimate the agent's motion in future novel task instances. We present empirical experiments using data gathered in our own lab and external corpora (VIRAT), based on which we find that our algorithm is not only efficiently implementable on a resource constrained platform but is also competitive in terms of accuracy with state-of-the-art alternatives (e.g., up to 20% better than the results reported in [1]).
---
paper_title: Novel planning-based algorithms for human motion prediction

paper_content:

Human motion prediction from visual tracking is a challenging problem with a wide array of applications such as robotics, video surveillance and situation understanding. Recently, planning-based methods -which assume that people move by planning over a cost function- have emerged as one of the most promising alternatives. Nevertheless, state of the art planning based algorithms have shortcomings regarding their computational complexity and ability to predict for arbitrary time intervals. This paper addresses these shortcomings by leveraging alternative planning techniques (Fast Marching Method) and formulating efficient algorithms for goal estimation and full spatiotemporal prediction with lower complexity than comparable approaches. In preliminary experiments, the proposed method significantly outperforms the accuracy of the current state-of-the-art approach while reducing the computation time by a factor of 30 using a parallel version of our algorithm.
---
paper_title: Goal-Directed Pedestrian Prediction

paper_content:

Recent advances in road safety have lead to a constant decline of injured traffic participants in Europe per year. Still, the number of injured pedestrians remains nearly constant. As a countermeasure, active pedestrian safety is the focus of current research, for which accurate pedestrian prediction is a prerequisite. In this scope, we propose a method for dynamics-and environment-based pedestrian prediction. We introduce the pedestrian's destination as a latent variable and thus convert the prediction problem into a planning problem. The planning is executed based on the current dynamics of the pedestrian. The distribution over the destinations is modeled using a Particle Filter. Experimental results show a significant improvement over existing approaches such as Kalman Filters.
---
paper_title: Activity forecasting

paper_content:

We address the task of inferring the future actions of people from noisy visual input. We denote this task activity forecasting. To achieve accurate activity forecasting, our approach models the effect of the physical environment on the choice of human actions. This is accomplished by the use of state-of-the-art semantic scene understanding combined with ideas from optimal control theory. Our unified model also integrates several other key elements of activity analysis, namely, destination forecasting, sequence smoothing and transfer learning. As proof-of-concept, we focus on the domain of trajectory-based activity analysis from visual input. Experimental results demonstrate that our model accurately predicts distributions over future actions of individuals. We show how the same techniques can improve the results of tracking algorithms by leveraging information about likely goals and trajectories.
---
paper_title: Road Infrastructure Indicators for Trajectory Prediction

paper_content:

Safe and comfortable path planning in a dynamic urban environment is essential to an autonomous vehicle. This requires the future trajectories of all other road users in the environment of the vehicle. These trajectories are predicted through modeling the motion and behaviour of these road users. In this work we state that for efficient trajectory prediction only motion indicators are not sufficient. Therefore, we propose using a curvilinear coordinate system with curvature as road infrastructure indicators to improve motion modeling and trajectory prediction. With experiments, we show that the curvilinear coordinate system with curvature sufficiently incorporates the road structure. Furthermore, we show that a sequence-tosequence RNN model is suitable to incorporate road curvature indicators directly into the modeling and prediction.
---
paper_title: 3DOF Pedestrian Trajectory Prediction Learned from Long-Term Autonomous Mobile Robot Deployment Data

paper_content:

This paper presents a novel 3DOF pedestrian trajectory prediction approach for autonomous mobile service robots. While most previously reported methods are based on learning of 2D positions in monocular camera images, our approach uses range-finder sensors to learn and predict 3DOF pose trajectories (i.e. 2D position plus 1D rotation within the world coordinate system). Our approach, T-Pose-LSTM (Temporal 3DOF-Pose Long-Short-Term Memory), is trained using long-term data from real-world robot deployments and aims to learn context-dependent (environment- and time-specific) human activities. Our approach incorporates long-term temporal information (i.e. date and time) with short-term pose observations as input. A sequence-to-sequence LSTM encoder-decoder is trained, which encodes observations into LSTM and then decodes as predictions. For deployment, it can perform on-the-fly prediction in real-time. Instead of using manually annotated data, we rely on a robust human detection, tracking and SLAM system, providing us with examples in a global coordinate system. We validate the approach using more than 15K pedestrian trajectories recorded in a care home environment over a period of three months. The experiment shows that the proposed T-Pose-LSTM model advances the state-of-the-art 2D-based method for human trajectory prediction in long-term mobile robot deployments.
---
paper_title: A mobile robot that understands pedestrian spatial behaviors

paper_content:

In human society, there are many invisible social rules or spatial effects existing in our environments. The robot that does not comprehend these spatial effects might harm people or itself. This paper presents a spatial behavior cognition model (SBCM) to describe the spatial effects existing between people and people, people and environments. By understanding the spatial effects in human-lived environments, the robot not only predicts pedestrian intentions and trajectories but also behaves socially acceptable motions. Moreover, the concept of pedestrian ego-graph (PEG) is proposed to efficiently query pedestrian-like paths for trajectory prediction. Model evaluation and experiments are shown to verify the proposed idea in this paper.
---
paper_title: Human Motion Prediction Under Social Grouping Constraints

paper_content:

Accurate long-term prediction of human motion in populated spaces is an important but difficult task for mobile robots and intelligent vehicles. What makes this task challenging is that human motion is influenced by a large variety of factors including the person's intention, the presence, attributes, actions, social relations and social norms of other surrounding agents, and the geometry and semantics of the environment. In this paper, we consider the problem of computing human motion predictions that account for such factors. We formulate the task as an MDP planning problem with stochastic policies and propose a weighted random walk algorithm in which each agent is locally influenced by social forces from other nearby agents. The novelty of this paper is that we incorporate social grouping information into the prediction process reflecting the soft formation constraints that groups typically impose to their members' motion. We show that our method makes more accurate predictions than three state-of-the-art methods in terms of probabilistic and geometrical performance metrics.
---
paper_title: Long-term vehicle motion prediction

paper_content:

Future driver assistance systems will have to cope with complex traffic situations, especially in the road crossing scenario. To detect potentially hazardous situations as early as possible, it is therefore desirable to know the position and motion of the ego-vehicle and vehicles around it for several seconds in advance. For this purpose, we propose in this study a long-term prediction approach based on a combined trajectory classification and particle filter framework. As a measure for the similarity between trajectories, we introduce the quaternion-based rotationally invariant longest common subsequence (QRLCS) metric. The trajectories are classified by a radial basis function (RBF) classifier with an architecture that is able to process trajectories of arbitrary non-uniform length. The particle filter framework simultaneously tracks and assesses a large number of motion hypotheses (∼102), where the class-specific probabilities estimated by the RBF classifier are used as a-priori probabilities for the hypotheses of the particle filter. The hypotheses are clustered with a mean-shift technique and are assigned a likelihood value. Motion prediction is performed based on the cluster centre with the highest likelihood. While traditional motion prediction based on curve radius and acceleration is inaccurate especially during turning manoeuvres, we show that our approach achieves a reasonable motion prediction even for long prediction intervals of 3 s for these complex motion patterns.
---
paper_title: Real-Time Certified Probabilistic Pedestrian Forecasting

paper_content:

The success of autonomous systems will depend upon their ability to safely navigate human-centric environments. This motivates the need for a real-time, probabilistic forecasting algorithm for pedestrians, cyclists, and other agents since these predictions will form a necessary step in assessing the risk of any action. This letter presents a novel approach to probabilistic forecasting for pedestrians based on weighted sums of ordinary differential equations that are learned from historical trajectory information within a fixed scene. The resulting algorithm is embarrassingly parallel and is able to work at real-time speeds using a naive Python implementation. The quality of predicted locations of agents generated by the proposed algorithm is validated on a variety of examples and is considerably higher than existing state of the art approaches over long time horizons.
---
paper_title: Intent-aware long-term prediction of pedestrian motion

paper_content:

We present a method to predict long-term motion of pedestrians, modeling their behavior as jump-Markov processes with their goal a hidden variable. Assuming approximately rational behavior, and incorporating environmental constraints and biases, including time-varying ones imposed by traffic lights, we model intent as a policy in a Markov decision process framework. We infer pedestrian state using a Rao-Blackwellized filter, and intent by planning according to a stochastic policy, reflecting individual preferences in aiming at the same goal.
---
paper_title: Multimodal Interaction-aware Motion Prediction for Autonomous Street Crossing

paper_content:

For mobile robots navigating on sidewalks, it is essential to be able to safely cross street intersections. Most existing approaches rely on the recognition of the traffic light signal to make an informed crossing decision. Although these approaches have been crucial enablers for urban navigation, the capabilities of robots employing such approaches are still limited to navigating only on streets containing signalized intersections. In this paper, we address this challenge and propose a multimodal convolutional neural network framework to predict the safety of a street intersection for crossing. Our architecture consists of two subnetworks; an interaction-aware trajectory estimation stream IA-TCNN, that predicts the future states of all observed traffic participants in the scene, and a traffic light recognition stream AtteNet. Our IA-TCNN utilizes dilated causal convolutions to model the behavior of the observable dynamic agents in the scene without explicitly assigning priorities to the interactions among them. While AtteNet utilizes Squeeze-Excitation blocks to learn a content-aware mechanism for selecting the relevant features from the data, thereby improving the noise robustness. Learned representations from the traffic light recognition stream are fused with the estimated trajectories from the motion prediction stream to learn the crossing decision. Furthermore, we extend our previously introduced Freiburg Street Crossing dataset with sequences captured at different types of intersections, demonstrating complex interactions among the traffic participants. Extensive experimental evaluations on public benchmark datasets and our proposed dataset demonstrate that our network achieves state-of-the-art performance for each of the subtasks, as well as for the crossing safety prediction.
---
paper_title: Bayesian intention inference for trajectory prediction with an unknown goal destination

paper_content:

Contextual cues can provide a rich source of information for robots that operate in the presence of other agents such as people, animals, vehicles and fellow robots. We are interested in context, in the form of the behavioural intent of an agent, for enhanced trajectory prediction. We present a Bayesian framework that estimates both the intended goal destination and future trajectory of a mobile agent moving among multiple static obstacles. Our method is based on multi-modal hypotheses of the intended goal, and is focused primarily on the long-term trajectory of the agent. We propose a computationally efficient solution and demonstrate its behaviour in a pedestrian scenario with a real-world data set. Results show the benefits of our method in comparison to traditional trajectory prediction methods and illustrate the feasibility of integration with higher-level planning algorithms.
---
paper_title: A Game-Theoretic Approach to Replanning-Aware Interactive Scene Prediction and Planning

paper_content:

This paper presents a novel cooperative-driving prediction and planning framework for dynamic environments based on the methods of game theory. The proposed algorithm can be used for highly automated driving on highways or as a sophisticated prediction module for advanced driver-assistance systems with no need for intervehicle communication. The main contribution of this paper is a model-based interaction-aware motion prediction of all vehicles in a scene. In contrast to other state-of-the-art approaches, the system also models the replanning capabilities of all drivers. With that, the driving strategy is able to capture complex interactions between vehicles, thus planning maneuver sequences over longer time horizons. It also enables an accurate prediction of traffic for the next immediate time step. The prediction model is supported by an interpretation of what other drivers intend to do, how they interact with traffic, and the ongoing observation. As part of the prediction loop, the proposed planning strategy incorporates the expected reactions of all traffic participants, offering cooperative and robust driving decisions. By means of experimental results under simulated highway scenarios, the validity of the proposed concept and its real-time capability is demonstrated.
---
paper_title: Predicting trajectories of vehicles using large-scale motion priors

paper_content:

We present a simple yet effective paradigm to accurately predict the future trajectories of observed vehicles in dense city environments. We equipped a large fleet of cars with cameras and performed city-scale structure-from-motion to accurately reconstruct 10M positions of their trajectories spanning over 1000h of driving.We demonstrate that this information can be used as a powerful high-fidelity prior to predict future trajectories of newly observed vehicles in the area without the need for any knowledge of road infrastructure or vehicle motion models. By relating the current position of the observed car to a large dataset of the previously exhibited motion in the area we can directly perform prediction of its future position.We evaluate our method on two large-scale data sets from San Francisco and New York City and demonstrate an order of magnitude improvement compared to a linear-motion based method. We also demonstrate that the performance naturally improves with the amount of data and ultimately yields a system that can accurately predict vehicle motion in challenging situations across extremes in traffic, time, and weather conditions.
---
paper_title: Pedestrian's Trajectory Forecast in Public Traffic with Artificial Neural Networks

paper_content:

This paper focuses on forecasting of pedestrian's short-time trajectories up to 2.5 s for traffic safety applications. We present a self-learning approach based on artificial neural network movement models and compare it to traditional constant velocity Kalman Filter prediction and extrapolation of polynomials fitted using a least-squares error. Trajectories of uninstructed pedestrians in public traffic at a real urban intersection are acquired by a wide angle stereo camera setup in combination with a 3D head tracking framework. Results using this real-world data show that the artificial neural network significantly improves forecast quality compared to other approaches especially for critical traffic scenes including velocity changes such as starting and stopping. For those velocity changes a reduction of position estimation errors of about 21% compared to the Kalman Filter and to extrapolation of polynomials is obtained. By means of a concrete pedestrian-vehicle scenario we demonstrate the benefit of the proposed approach for an advanced driver assistant system in terms of reaction time.
---
paper_title: Will the Pedestrian Cross? A Study on Pedestrian Path Prediction

paper_content:

Future vehicle systems for active pedestrian safety will not only require a high recognition performance but also an accurate analysis of the developing traffic situation. In this paper, we present a study on pedestrian path prediction and action classification at short subsecond time intervals. We consider four representative approaches: two novel approaches (based on Gaussian process dynamical models and probabilistic hierarchical trajectory matching) that use augmented features derived from dense optical flow and two approaches as baseline that use positional information only (a Kalman filter and its extension to interacting multiple models). In experiments using stereo vision data obtained from a vehicle, we investigate the accuracy of path prediction and action classification at various time horizons, the effect of various errors (image localization, vehicle egomotion estimation), and the benefit of the proposed approaches. The scenario of interest is that of a crossing pedestrian, who might stop or continue walking at the road curbside. Results indicate similar performance of the four approaches on walking motion, with near-linear dynamics. During stopping, however, the two newly proposed approaches, with nonlinear and/or higher order models and augmented motion features, achieve a more accurate position prediction of 10-50 cm at a time horizon of 0-0.77 s around the stopping event.
---
paper_title: Incremental Learning of Statistical Motion Patterns With Growing Hidden Markov Models

paper_content:

Modeling and predicting human and vehicle motion is an active research do- main. Due to the difﬁculty of modeling the various factors that determine motion (eg in- ternal state, perception, etc.) this is often tackled by applying machine learning techniques to build a statistical model, using as input a collection of trajectories gathered through a sensor (eg camera, laser scanner), and then using that model to predict further motion. Un- fortunately, most current techniques use off-line learning algorithms, meaning that they are not able to learn new motion patterns once the learning stage has ﬁnished. In this paper, we present an approach which is able to learn new motion patterns incrementally, and in parallel with prediction. Our work is based on a novel extension to Hidden Markov Models called Growing Hidden Markov models.
---
paper_title: GLMP- realtime pedestrian path prediction using global and local movement patterns

paper_content:

We present a novel real-time algorithm to predict the path of pedestrians in cluttered environments. Our approach makes no assumption about pedestrian motion or crowd density, and is useful for short-term as well as long-term prediction. We interactively learn the characteristics of pedestrian motion and movement patterns from 2D trajectories using Bayesian inference. These include local movement patterns corresponding to the current and preferred velocities and global characteristics such as entry points and movement features. Our approach involves no precomputation and we demonstrate the real-time performance of our prediction algorithm on sparse and noisy trajectory data extracted from dense indoor and outdoor crowd videos. The combination of local and global movement patterns can improve the accuracy of long-term prediction by 12–18% over prior methods in high-density videos.
---
paper_title: Augmented dictionary learning for motion prediction

paper_content:

Developing accurate models and efficient representations of multivariate trajectories is important for understanding the behavior patterns of mobile agents. This work presents a dictionary learning algorithm for developing a part-based trajectory representation, which combines merits of the existing Markovian-based and clustering-based approaches. In particular, this work presents the augmented semi-nonnegative sparse coding (ASNSC) algorithm for solving a constrained dictionary learning problem, and shows that the proposed method would converge to a local optimum given a convexity condition. We consider a trajectory modeling application, in which the learned dictionary atoms correspond to local motion patterns. Classical semi-nonnegative sparse coding approaches would add dictionary atoms with opposite signs to reduce the representational error, which can lead to learning noisy dictionary atoms that correspond poorly to local motion patterns. ASNSC addresses this problem and learns a concise set of intuitive motion patterns. ASNSC shows significant improvement over existing trajectory modeling methods in both prediction accuracy and computational time, as revealed by extensive numerical analysis on real datasets.
---
paper_title: The Simpler the Better: Constant Velocity for Pedestrian Motion Prediction.

paper_content:

Pedestrian motion prediction is a fundamental task for autonomous robots and vehicles to operate safely. In recent years many complex models have been proposed to address this problem. While complex models can be justified, simple models should be preferred given the same or better performance. In this work we show that a simple Constant Velocity Model can achieve competitive performance on this task. We evaluate the Constant Velocity Model using two popular benchmark datasets for pedestrian motion prediction and show that it outperforms state-of-the-art models and several common baselines. The success of this model indicates that either neural networks are not able to make use of the additional information they are provided with, or it is not as relevant as commonly believed. Therefore, we analyze how neural networks process this information and how it impacts their predictions. Our analysis shows that neural networks implicitly learn environmental priors that have a negative impact on their generalization capability, most of the pedestrian's motion history is ignored and interactions - while happening - are too complex to predict. These findings explain the success of the Constant Velocity Model and lead to a better understanding of the problem at hand.
---
paper_title: Probabilistic Map-based Pedestrian Motion Prediction Taking Traffic Participants into Consideration

paper_content:

As pedestrians are one of the most vulnerable traffic participants, their motion prediction is of utmost importance for intelligent transportation systems. Predicting motions of pedestrians is especially hard since they move in less structured environments and have less inertia compared to road vehicles. To account for this uncertainty, we present an approach for probabilistic prediction of pedestrian motion using Markov chains. In contrast to previous work, we not only consider motion models, constraints from a semantic map, and various goals, but also explicitly adapt the prediction based on crash probabilities with other traffic participants. Also, our approach works in any situation; this is typically challenging for pure machine learning techniques that learn behaviors for a particular road section and which might consequently struggle with a different road section. The usefulness of combining the aforementioned aspects in a single approach is demonstrated by an evaluation using recordings of real pedestrians.
---
paper_title: Multi-Agent Tensor Fusion for Contextual Trajectory Prediction

paper_content:

Accurate prediction of others' trajectories is essential for autonomous driving. Trajectory prediction is challenging because it requires reasoning about agents' past movements, social interactions among varying numbers and kinds of agents, constraints from the scene context, and the stochasticity of human behavior. Our approach models these interactions and constraints jointly within a novel Multi-Agent Tensor Fusion (MATF) network. Specifically, the model encodes multiple agents' past trajectories and the scene context into a Multi-Agent Tensor, then applies convolutional fusion to capture multiagent interactions while retaining the spatial structure of agents and the scene context. The model decodes recurrently to multiple agents' future trajectories, using adversarial loss to learn stochastic predictions. Experiments on both highway driving and pedestrian crowd datasets show that the model achieves state-of-the-art prediction accuracy.
---
paper_title: Pedestrian path prediction using body language traits

paper_content:

Driver Assistance Systems have achieved a high level of maturity in the latest years. As an example of that, sophisticated pedestrian protection systems are already available in a number of commercial vehicles from several OEMs. However, accurate pedestrian path prediction is needed in order to go a step further in terms of safety and reliability, since it can make the difference between effective and non-effective intervention. In this paper, we consider the three-dimensional pedestrian body language in order to perform path prediction in a probabilistic framework. For this purpose, the different body parts and joints are detected using stereo vision. We propose the use of GPDM (Gaussian Process Dynamical Models) for reducing the high dimensionality of the input feature vector (composed by joints and displacement vectors) in the 3D pose space and for learning the pedestrian dynamics in a latent space. Experimental results show that accurate path prediction can be achieved at a time horizon of ≈ 0.8 s.
---
paper_title: Goal-Directed Pedestrian Prediction

paper_content:

Recent advances in road safety have lead to a constant decline of injured traffic participants in Europe per year. Still, the number of injured pedestrians remains nearly constant. As a countermeasure, active pedestrian safety is the focus of current research, for which accurate pedestrian prediction is a prerequisite. In this scope, we propose a method for dynamics-and environment-based pedestrian prediction. We introduce the pedestrian's destination as a latent variable and thus convert the prediction problem into a planning problem. The planning is executed based on the current dynamics of the pedestrian. The distribution over the destinations is modeled using a Particle Filter. Experimental results show a significant improvement over existing approaches such as Kalman Filters.
---
paper_title: Predicting actions to act predictably: Cooperative partial motion planning with maximum entropy models

paper_content:

This paper reports on a data-driven motion planning approach for interaction-aware, socially-compliant robot navigation among human agents. Autonomous mobile robots navigating in workspaces shared with human agents require motion planning techniques providing seamless integration and smooth navigation in such. Smooth integration in mixed scenarios calls for two abilities of the robot: predicting actions of others and acting predictably for them. The former requirement requests trainable models of agent behaviors in order to accurately forecast their actions in the future, taking into account their reaction on the robot's decisions. A human-like navigation style of the robot facilitates other agents—most likely not aware of the underlying planning technique applied—to predict the robot motion vice versa, resulting in smoother joint navigation. The approach presented in this paper is based on a feature-based maximum entropy model and is able to guide a robot in an unstructured, real-world environment. The model is trained to predict joint behavior of heterogeneous groups of agents from onboard data of a mobile platform. We evaluate the benefit of interaction-aware motion planning in a realistic public setting with a total distance traveled of over 4 km. Interestingly the motion models learned from human-human interaction did not hold for robot-human interaction, due to the high attention and interest of pedestrians in testing basic braking functionality of the robot.
---
paper_title: Predicting wide receiver trajectories in American football

paper_content:

Predicting the trajectory of a wide receiver in the game of American football requires prior knowledge about the game (e.g., route trees, defensive formations) and an accurate model of how the environment will change over time (e.g., opponent reaction strategies, motion attributes of players). Our aim is to build a computational model of the wide receiver, which takes into account prior knowledge about the game and short-term predictive models of how the environment will change over time. While prior knowledge of the game is readily accessible, it is quite challenging to build predictive models of how the environment will change over time. We propose several models for predicting short-term motions of opponent players to generate dynamic input features for our wide receiver forecasting model. In particular, we model the wide receiver with a Markov Decision Process (MDP), where the reward function is a linear combination of static features (prior knowledge about the game) and dynamic features (short-term prediction of opponent players). Since the dynamic features change over time, we make recursive calls to an inference procedure over the MDP while updating the dynamic features. We validate our technique on a video dataset of American football plays. Our results show that more informed models that accurately predict the motions of the defensive players are better at forecasting wide receiver plays.
---
paper_title: Building Prior Knowledge: A Markov Based Pedestrian Prediction Model Using Urban Environmental Data

paper_content:

Autonomous Vehicles navigating in urban areas have a need to understand and predict future pedestrian behavior for safer navigation. This high level of situational awareness requires observing pedestrian behavior and extrapolating their positions to know future positions. While some work has been done in this field using Hidden Markov Models (HMMs), one of the few observed drawbacks of the method is the need for informed priors for learning behavior. In this work, an extension to the Growing Hidden Markov Model (GHMM) method is proposed to solve some of these drawbacks. This is achieved by building on existing work using potential cost maps and the principle of Natural Vision. As a consequence, the proposed model is able to predict pedestrian positions more precisely over a longer horizon compared to the state of the art. The method is tested over "legal" and "illegal" behavior of pedestrians, having trained the model with sparse observations and partial trajectories. The method, with no training data, is compared against a trained state of the art model. It is observed that the proposed method is robust even in new, previously unseen areas.
---
paper_title: A probabilistic model of human motion and navigation intent for mobile robot path planning

paper_content:

In order to effectively plan paths in environments inhabited by humans, robots must accurately predict human motion. Typical approaches to human prediction simply assume a constant velocity which is not always valid. This paper proposes to determine the likely navigation intent of humans and use that to predict human motion. Navigation intent is determined by the function and structure of the environment. Manually assigned functional places are combined with automatically extracted navigation way-points to define a number of likely navigation targets within the environment. To predict human motion toward these targets, a probabilistic model of human motion is proposed which is based on motion probability grids generated from observed motion. The models of human navigation intent and motion are integrated with an autonomous mobile robot system, with a laser range sensor detecting humans moving within the environment, and a path planning system. The models of human navigation intent and motion are verified using real captured human motion data from an office environment. Examples of human motion prediction are also presented.
---
paper_title: Activity forecasting

paper_content:

We address the task of inferring the future actions of people from noisy visual input. We denote this task activity forecasting. To achieve accurate activity forecasting, our approach models the effect of the physical environment on the choice of human actions. This is accomplished by the use of state-of-the-art semantic scene understanding combined with ideas from optimal control theory. Our unified model also integrates several other key elements of activity analysis, namely, destination forecasting, sequence smoothing and transfer learning. As proof-of-concept, we focus on the domain of trajectory-based activity analysis from visual input. Experimental results demonstrate that our model accurately predicts distributions over future actions of individuals. We show how the same techniques can improve the results of tracking algorithms by leveraging information about likely goals and trajectories.
---
paper_title: A Data-driven Model for Interaction-Aware Pedestrian Motion Prediction in Object Cluttered Environments

paper_content:

This paper reports on a data-driven, interaction-aware motion prediction approach for pedestrians in environments cluttered with static obstacles. When navigating in such workspaces shared with humans, robots need accurate motion predictions of the surrounding pedestrians. Human navigation behavior is mostly influenced by their surrounding pedestrians and by the static obstacles in their vicinity. In this paper we introduce a new model based on Long-Short Term Memory (LSTM) neural networks, which is able to learn human motion behavior from demonstrated data. To the best of our knowledge, this is the first approach using LSTMs, that incorporates both static obstacles and surrounding pedestrians for trajectory forecasting. As part of the model, we introduce a new way of encoding surrounding pedestrians based on a 1d-grid in polar angle space. We evaluate the benefit of interaction-aware motion prediction and the added value of incorporating static obstacles on both simulation and real-world datasets by comparing with state-of-the-art approaches. The results show, that our new approach outperforms the other approaches while being very computationally efficient and that taking into account static obstacles for motion predictions significantly improves the prediction accuracy, especially in cluttered environments.
---
paper_title: Multi-Modal Trajectory Prediction of Surrounding Vehicles with Maneuver based LSTMs

paper_content:

To safely and efficiently navigate through complex traffic scenarios, autonomous vehicles need to have the ability to predict the future motion of surrounding vehicles. Multiple interacting agents, the multi-modal nature of driver behavior, and the inherent uncertainty involved in the task make motion prediction of surrounding vehicles a challenging problem. In this paper, we present an LSTM model for interaction aware motion prediction of surrounding vehicles on freeways. Our model assigns confidence values to maneuvers being performed by vehicles and outputs a multi-modal distribution over future motion based on them. We compare our approach with the prior art for vehicle motion prediction on the publicly available NGSIM US-101 and I-80 datasets. Our results show an improvement in terms of RMS values of prediction error. We also present an ablative analysis of the components of our proposed model and analyze the predictions made by the model in complex traffic scenarios.
---
paper_title: Using road topology to improve cyclist path prediction

paper_content:

We learn motion models for cyclist path prediction on real-world tracks obtained from a moving vehicle, and propose to exploit the local road topology to obtain better predictive distributions. The tracks are extracted from the Tsinghua-Daimler Cyclist Benchmark for cyclist detection, and corrected for vehicle egomotion. Tracks are then spatially aligned to local curves and crossings in the road. We study a standard approach for path prediction in the literature based on Kalman Filters, as well as a mixture of specialized filters related to specific road orientations at junctions. Our experiments demonstrate an improved prediction accuracy (up to 20% on sharp turns) of mixing specialized motion models for canonical directions, and prior knowledge on the road topology. The new track data complements the existing video, disparity and annotation data of the original benchmark, and will be made publicly available.
---
paper_title: The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems

paper_content:

Scenario-based testing for the safety validation of highly automated vehicles is a promising approach that is being examined in research and industry. This approach heavily relies on data from real-world scenarios to derive the necessary scenario information for testing. Measurement data should be collected at a reasonable effort, contain naturalistic behavior of road users and include all data relevant for a description of the identified scenarios in sufficient quality. However, the current measurement methods fail to meet at least one of the requirements. Thus, we propose a novel method to measure data from an aerial perspective for scenario-based validation fulfilling the mentioned requirements. Furthermore, we provide a large-scale naturalistic vehicle trajectory dataset from German highways called highD. We evaluate the data in terms of quantity, variety and contained scenarios. Our dataset consists of 16.5 hours of measurements from six locations with 110 000 vehicles, a total driven distance of 45 000 km and 5600 recorded complete lane changes. The highD dataset is available online at: http://www.highD-dataset.com
---
paper_title: A fast marching level set method for monotonically advancing fronts.

paper_content:

Abstract ::: A fast marching level set method is presented for monotonically advancing fronts, which leads to an extremely fast scheme for solving the Eikonal equation. Level set methods are numerical techniques for computing the position of propagating fronts. They rely on an initial value partial differential equation for a propagating level set function and use techniques borrowed from hyperbolic conservation laws. Topological changes, corner and cusp development, and accurate determination of geometric properties such as curvature and normal direction are naturally obtained in this setting. This paper describes a particular case of such methods for interfaces whose speed depends only on local position. The technique works by coupling work on entropy conditions for interface motion, the theory of viscosity solutions for Hamilton-Jacobi equations, and fast adaptive narrow band level set methods. The technique is applicable to a variety of problems, including shape-from-shading problems, lithographic development calculations in microchip manufacturing, and arrival time problems in control theory.
---
paper_title: Conditional transition maps: Learning motion patterns in dynamic environments

paper_content:

In this paper we introduce a method for learning motion patterns in dynamic environments. Representations of dynamic environments have recently received an increasing amount of attention in the research community. Understanding dynamic environments is seen as one of the key challenges in order to enable autonomous navigation in real-world scenarios. However, representing the temporal dimension is a challenge yet to be solved. In this paper we introduce a spatial representation, which encapsulates the statistical dynamic behavior observed in the environment. The proposed Conditional Transition Map (CTMap) is a grid-based representation that associates a probability distribution for an object exiting the cell, given its entry direction. The transition parameters are learned from a temporal signal of occupancy on cells by using a local-neighborhood cross-correlation method. In this paper, we introduce the CTMap, the learning approach and present a proof-of-concept method for estimating future paths of dynamic objects, called Conditional Probability Propagation Tree (CPPTree). The evaluation is done using a real-world dataset collected at a busy roundabout.
---
paper_title: Context-Aware Trajectory Prediction

paper_content:

Human motion and behaviour in crowded spaces is influenced by several factors, such as the dynamics of other moving agents in the scene, as well as the static elements that might be perceived as points of attraction or obstacles. In this work, we present a new model for human trajectory prediction which is able to take advantage of both human-human and human-space interactions. The future trajectory of humans, are generated by observing their past positions and interactions with the surroundings. To this end, we propose a"context-aware"recurrent neural network LSTM model, which can learn and predict human motion in crowded spaces such as a sidewalk, a museum or a shopping mall. We evaluate our model on a public pedestrian datasets, and we contribute a new challenging dataset that collects videos of humans that navigate in a (real) crowded space such as a big museum. Results show that our approach can predict human trajectories better when compared to previous state-of-the-art forecasting models.
---
paper_title: Unfreezing the robot: Navigation in dense, interacting crowds

paper_content:

In this paper, we study the safe navigation of a mobile robot through crowds of dynamic agents with uncertain trajectories. Existing algorithms suffer from the “freezing robot” problem: once the environment surpasses a certain level of complexity, the planner decides that all forward paths are unsafe, and the robot freezes in place (or performs unnecessary maneuvers) to avoid collisions. Since a feasible path typically exists, this behavior is suboptimal. Existing approaches have focused on reducing the predictive uncertainty for individual agents by employing more informed models or heuristically limiting the predictive covariance to prevent this overcautious behavior. In this work, we demonstrate that both the individual prediction and the predictive uncertainty have little to do with the frozen robot problem. Our key insight is that dynamic agents solve the frozen robot problem by engaging in “joint collision avoidance”: They cooperatively make room to create feasible trajectories. We develop IGP, a nonparametric statistical model based on dependent output Gaussian processes that can estimate crowd interaction from data. Our model naturally captures the non-Markov nature of agent trajectories, as well as their goal-driven navigation. We then show how planning in this model can be efficiently implemented using particle based inference. Lastly, we evaluate our model on a dataset of pedestrians entering and leaving a building, first comparing the model with actual pedestrians, and find that the algorithm either outperforms human pedestrians or performs very similarly to the pedestrians. We also present an experiment where a covariance reduction method results in highly overcautious behavior, while our model performs desirably.
---
paper_title: Sampling-based algorithms for optimal motion planning

paper_content:

United States. Air Force Office of Scientific Research (Michigan/AFRL Collaborative Center in Control Science Grant FA 8650-07-2-3744)
---
paper_title: Social force model for pedestrian dynamics

paper_content:

It is suggested that the motion of pedestrians can be described as if they would be subject to ``social forces.'' These ``forces'' are not directly exerted by the pedestrians' personal environment, but they are a measure for the internal motivations of the individuals to perform certain actions (movements). The corresponding force concept is discussed in more detail and can also be applied to the description of other behaviors. In the presented model of pedestrian behavior several force terms are essential: first, a term describing the acceleration towards the desired velocity of motion; second, terms reflecting that a pedestrian keeps a certain distance from other pedestrians and borders; and third, a term modeling attractive effects. The resulting equations of motion of nonlinearly coupled Langevin equations. Computer simulations of crowds of interacting pedestrians show that the social force model is capable of describing the self-organization of several observed collective effects of pedestrian behavior very realistically.
---
paper_title: Planning-based prediction for pedestrians

paper_content:

We present a novel approach for determining robot movements that efficiently accomplish the robot's tasks while not hindering the movements of people within the environment. Our approach models the goal-directed trajectories of pedestrians using maximum entropy inverse optimal control. The advantage of this modeling approach is the generality of its learned cost function to changes in the environment and to entirely different environments. We employ the predictions of this model of pedestrian trajectories in a novel incremental planner and quantitatively show the improvement in hindrance-sensitive robot trajectory planning provided by our approach.
---
paper_title: Human Motion Prediction Under Social Grouping Constraints

paper_content:

Accurate long-term prediction of human motion in populated spaces is an important but difficult task for mobile robots and intelligent vehicles. What makes this task challenging is that human motion is influenced by a large variety of factors including the person's intention, the presence, attributes, actions, social relations and social norms of other surrounding agents, and the geometry and semantics of the environment. In this paper, we consider the problem of computing human motion predictions that account for such factors. We formulate the task as an MDP planning problem with stochastic policies and propose a weighted random walk algorithm in which each agent is locally influenced by social forces from other nearby agents. The novelty of this paper is that we incorporate social grouping information into the prediction process reflecting the soft formation constraints that groups typically impose to their members' motion. We show that our method makes more accurate predictions than three state-of-the-art methods in terms of probabilistic and geometrical performance metrics.
---
paper_title: Intent-aware long-term prediction of pedestrian motion

paper_content:

We present a method to predict long-term motion of pedestrians, modeling their behavior as jump-Markov processes with their goal a hidden variable. Assuming approximately rational behavior, and incorporating environmental constraints and biases, including time-varying ones imposed by traffic lights, we model intent as a policy in a Markov decision process framework. We infer pedestrian state using a Rao-Blackwellized filter, and intent by planning according to a stochastic policy, reflecting individual preferences in aiming at the same goal.
---
paper_title: Forecasting Interactive Dynamics of Pedestrians with Fictitious Play

paper_content:

We develop predictive models of pedestrian dynamics by encoding the coupled nature of multi-pedestrian interaction using game theory, and deep learning-based visual analysis to estimate person-specific behavior parameters. Building predictive models for multi-pedestrian interactions however, is very challenging due to two reasons: (1) the dynamics of interaction are complex interdependent processes, where the predicted behavior of one pedestrian can affect the actions taken by others and (2) dynamics are variable depending on an individuals physical characteristics (e.g., an older person may walk slowly while the younger person may walk faster). To address these challenges, we (1) utilize concepts from game theory to model the interdependent decision making process of multiple pedestrians and (2) use visual classifiers to learn a mapping from pedestrian appearance to behavior parameters. We evaluate our proposed model on several public multiple pedestrian interaction video datasets. Results show that our strategic planning model explains human interactions 25% better when compared to state-of-the-art methods.
---
paper_title: Long-term Path Prediction in Urban Scenarios using Circular Distributions

paper_content:

Abstract Human ability to foresee the near future plays a key role in everyone's life to prevent potentially dangerous situations. To be able to make predictions is crucial when people have to interact with the surrounding environment. Modeling such capability can lead to the design of automated warning systems and provide moving robots with an intelligent way of interaction with changing situation. In this work we focus on a typical urban human-scene where we aim at predicting an agent's behavior using a stochastic model. In this approach, we fuse the various factors that would contribute to a human motion in different contexts. Our method uses previously observed trajectories to build point-wise circular distributions that after combination, provide a statistical smooth prediction towards the most likely areas. More specifically, a ray-launching procedure, based on a semantic segmentation, gives a coarse scene representation for collision avoidance; a nearly-constant velocity dynamic model smooths the acceleration progression and knowledge of the agent's destination may further steer the path prediction. Experimental results in structured scenes, validate the effectiveness of the method in predicting paths in comparison to actual trajectories.
---
paper_title: On the Complexity of Solving Markov Decision Problems

paper_content:

Markov decision problems (MDPs) provide the foundations for a number of problems of interest to AI researchers studying automated planning and reinforcement learning. In this paper, we summarize results regarding the complexity of solving MDPs and the running time of MDP solution algorithms. We argue that, although MDPs can be solved efficiently in theory, more study is needed to reveal practical algorithms for solving large problems quickly. To encourage future research, we sketch some alternative methods of analysis that rely on the structure of MDPs.
---
paper_title: The Simpler the Better: Constant Velocity for Pedestrian Motion Prediction.

paper_content:

Pedestrian motion prediction is a fundamental task for autonomous robots and vehicles to operate safely. In recent years many complex models have been proposed to address this problem. While complex models can be justified, simple models should be preferred given the same or better performance. In this work we show that a simple Constant Velocity Model can achieve competitive performance on this task. We evaluate the Constant Velocity Model using two popular benchmark datasets for pedestrian motion prediction and show that it outperforms state-of-the-art models and several common baselines. The success of this model indicates that either neural networks are not able to make use of the additional information they are provided with, or it is not as relevant as commonly believed. Therefore, we analyze how neural networks process this information and how it impacts their predictions. Our analysis shows that neural networks implicitly learn environmental priors that have a negative impact on their generalization capability, most of the pedestrian's motion history is ignored and interactions - while happening - are too complex to predict. These findings explain the success of the Constant Velocity Model and lead to a better understanding of the problem at hand.
---
paper_title: Deterministic Sampling-Based Motion Planning: Optimality, Complexity, and Performance

paper_content:

Probabilistic sampling-based algorithms, such as the probabilistic roadmap (PRM) and the rapidly-exploring random tree (RRT) algorithms, represent one of the most successful approaches to robotic motion planning, due to their strong theoretical properties (in terms of probabilistic completeness or even asymptotic optimality) and remarkable practical performance. Such algorithms are probabilistic in that they compute a path by connecting independent and identically distributed (i.i.d.) random points in the configuration space. Their randomization aspect, however, makes several tasks challenging, including certification for safety-critical applications and use of offline computation to improve real-time execution. Hence, an important open question is whether similar (or better) theoretical guarantees and practical performance could be obtained by considering deterministic, as opposed to random sampling sequences. The objective of this paper is to provide a rigorous answer to this question. The focus is on the PRM algorithm—our results, however, generalize to other batch-processing algorithms such as \(\text {FMT}^*\). Specifically, we first show that PRM, for a certain selection of tuning parameters and deterministic low-dispersion sampling sequences, is deterministically asymptotically optimal, i.e., it returns a path whose cost converges deterministically to the optimal one as the number of points goes to infinity. Second, we characterize the convergence rate, and we find that the factor of sub-optimality can be very explicitly upper-bounded in terms of the \(\ell _2\)-dispersion of the sampling sequence and the connection radius of PRM. Third, we show that an asymptotically optimal version of PRM exists with computational and space complexity arbitrarily close to O(n) (the theoretical lower bound), where n is the number of points in the sequence. This is in stark contrast to the \(O(n\, \log n)\) complexity results for existing asymptotically-optimal probabilistic planners. Finally, through numerical experiments, we show that planning with deterministic low-dispersion sampling generally provides superior performance in terms of path cost and success rate.
---
paper_title: Learning Motion Patterns of People for Compliant Robot Motion

paper_content:

Whenever people move through their environments they do not move randomly. Instead, they usually follow specific trajectories or motion patterns corresponding to their intentions. Knowledge about such patterns enables a mobile robot to robustly keep track of persons in its environment and to improve its behavior. In this paper we propose a technique for learning collections of trajectories that characterize typical motion patterns of persons. Data recorded with laser-range finders are clustered using the expectation maximization algorithm. Based on the result of the clustering process, we derive a hidden Markov model that is applied to estimate the current and future positions of persons based on sensory input. We also describe how to incorporate the probabilistic belief about the potential trajectories of persons into the path planning process of a mobile robot. We present several experiments carried out in different environments with a mobile robot equipped with a laser-range scanner and a camera system...
---
paper_title: Pedestrian Behavior Prediction based on Motion Patterns for Vehicle-to-Pedestrian Collision Avoidance

paper_content:

This paper proposes a prediction method for vehicle-to-pedestrian collision avoidance, which learns and then predicts pedestrian behaviors as their motion instances are being observed. During learning, known trajectories are clustered to form motion patterns (MP), which become knowledge a priori to a multi-level prediction model that predicts long-term or short-term pedestrian behaviors. Simulation results show that it works well in a complex structured environment and the prediction is consistent with actual behaviors.
---
paper_title: Social LSTM: Human Trajectory Prediction in Crowded Spaces

paper_content:

Pedestrians follow different trajectories to avoid obstacles and accommodate fellow pedestrians. Any autonomous vehicle navigating such a scene should be able to foresee the future positions of pedestrians and accordingly adjust its path to avoid collisions. This problem of trajectory prediction can be viewed as a sequence generation task, where we are interested in predicting the future trajectory of people based on their past positions. Following the recent success of Recurrent Neural Network (RNN) models for sequence prediction tasks, we propose an LSTM model which can learn general human movement and predict their future trajectories. This is in contrast to traditional approaches which use hand-crafted functions such as Social forces. We demonstrate the performance of our method on several public datasets. Our model outperforms state-of-the-art methods on some of these datasets. We also analyze the trajectories predicted by our model to demonstrate the motion behaviour learned by our model.
---
paper_title: Driver and pedestrian awareness-based collision risk analysis

paper_content:

We present a novel approach for vehicle-pedestrian collision risk analysis that incorporates mutual situational awareness, a degree of potential motion coupling and the spatial layout of the environment. The approach uses a Dynamic Bayesian Network (DBN) for modeling the individual object paths; collision risk is subsequently computed by an intersection operation.
---
paper_title: Social Attention: Modeling Attention in Human Crowds

paper_content:

Robots that navigate through human crowds need to be able to plan safe, efficient, and human predictable trajectories. This is a particularly challenging problem as it requires the robot to predict future human trajectories within a crowd where everyone implicitly cooperates with each other to avoid collisions. Previous approaches to human trajectory prediction have modeled the interactions between humans as a function of proximity. However, that is not necessarily true as some people in our immediate vicinity moving in the same direction might not be as important as other people that are further away, but that might collide with us in the future. In this work, we propose Social Attention, a novel trajectory prediction model that captures the relative importance of each person when navigating in the crowd, irrespective of their proximity. We demonstrate the performance of our method against a state-of-the-art approach on two publicly available crowd datasets and analyze the trained attention model to gain a better understanding of which surrounding agents humans attend to, when navigating in a crowd.
---
paper_title: Activity forecasting

paper_content:

We address the task of inferring the future actions of people from noisy visual input. We denote this task activity forecasting. To achieve accurate activity forecasting, our approach models the effect of the physical environment on the choice of human actions. This is accomplished by the use of state-of-the-art semantic scene understanding combined with ideas from optimal control theory. Our unified model also integrates several other key elements of activity analysis, namely, destination forecasting, sequence smoothing and transfer learning. As proof-of-concept, we focus on the domain of trajectory-based activity analysis from visual input. Experimental results demonstrate that our model accurately predicts distributions over future actions of individuals. We show how the same techniques can improve the results of tracking algorithms by leveraging information about likely goals and trajectories.
---
paper_title: A Data-driven Model for Interaction-Aware Pedestrian Motion Prediction in Object Cluttered Environments

paper_content:

This paper reports on a data-driven, interaction-aware motion prediction approach for pedestrians in environments cluttered with static obstacles. When navigating in such workspaces shared with humans, robots need accurate motion predictions of the surrounding pedestrians. Human navigation behavior is mostly influenced by their surrounding pedestrians and by the static obstacles in their vicinity. In this paper we introduce a new model based on Long-Short Term Memory (LSTM) neural networks, which is able to learn human motion behavior from demonstrated data. To the best of our knowledge, this is the first approach using LSTMs, that incorporates both static obstacles and surrounding pedestrians for trajectory forecasting. As part of the model, we introduce a new way of encoding surrounding pedestrians based on a 1d-grid in polar angle space. We evaluate the benefit of interaction-aware motion prediction and the added value of incorporating static obstacles on both simulation and real-world datasets by comparing with state-of-the-art approaches. The results show, that our new approach outperforms the other approaches while being very computationally efficient and that taking into account static obstacles for motion predictions significantly improves the prediction accuracy, especially in cluttered environments.
---
paper_title: Behavior estimation for a complete framework for human motion prediction in crowded environments

paper_content:

In the present work, we propose and validate a complete probabilistic framework for human motion prediction in urban or social environments. Additionally, we formulate a powerful and useful tool: the human motion behavior estimator. Three different basic behaviors have been detected: Aware, Balanced and Unaware. Our approach is based on the Social Force Model (SFM) and the intentionality prediction BHMIP. The main contribution of the present work is to make use of the behavior estimator for formulating a reliable prediction framework of human trajectories under the influence of dynamic crowds, robots, and in general any moving obstacle. Accordingly, we have demonstrated the great performance of our long-term prediction algorithm, in real scenarios, comparing to other prediction methods.
---
paper_title: Unfreezing the robot: Navigation in dense, interacting crowds

paper_content:

In this paper, we study the safe navigation of a mobile robot through crowds of dynamic agents with uncertain trajectories. Existing algorithms suffer from the “freezing robot” problem: once the environment surpasses a certain level of complexity, the planner decides that all forward paths are unsafe, and the robot freezes in place (or performs unnecessary maneuvers) to avoid collisions. Since a feasible path typically exists, this behavior is suboptimal. Existing approaches have focused on reducing the predictive uncertainty for individual agents by employing more informed models or heuristically limiting the predictive covariance to prevent this overcautious behavior. In this work, we demonstrate that both the individual prediction and the predictive uncertainty have little to do with the frozen robot problem. Our key insight is that dynamic agents solve the frozen robot problem by engaging in “joint collision avoidance”: They cooperatively make room to create feasible trajectories. We develop IGP, a nonparametric statistical model based on dependent output Gaussian processes that can estimate crowd interaction from data. Our model naturally captures the non-Markov nature of agent trajectories, as well as their goal-driven navigation. We then show how planning in this model can be efficiently implemented using particle based inference. Lastly, we evaluate our model on a dataset of pedestrians entering and leaving a building, first comparing the model with actual pedestrians, and find that the algorithm either outperforms human pedestrians or performs very similarly to the pedestrians. We also present an experiment where a covariance reduction method results in highly overcautious behavior, while our model performs desirably.
---
paper_title: Behavioral Priors for Detection and Tracking of Pedestrians in Video Sequences

paper_content:

In this paper we address the problem of detection and tracking of pedestrians in complex scenarios. The inclusion of prior knowledge is more and more crucial in scene analysis to guarantee flexibility and robustness, necessary to have reliability in complex scenes. We aim to combine image processing methods with behavioral models of pedestrian dynamics, calibrated on real data. We introduce Discrete Choice Models (DCM) for pedestrian behavior and we discuss their integration in a detection and tracking context. The obtained results show how it is possible to combine both methodologies to improve the performances of such systems in complex sequences.
---
paper_title: 3DOF Pedestrian Trajectory Prediction Learned from Long-Term Autonomous Mobile Robot Deployment Data

paper_content:

This paper presents a novel 3DOF pedestrian trajectory prediction approach for autonomous mobile service robots. While most previously reported methods are based on learning of 2D positions in monocular camera images, our approach uses range-finder sensors to learn and predict 3DOF pose trajectories (i.e. 2D position plus 1D rotation within the world coordinate system). Our approach, T-Pose-LSTM (Temporal 3DOF-Pose Long-Short-Term Memory), is trained using long-term data from real-world robot deployments and aims to learn context-dependent (environment- and time-specific) human activities. Our approach incorporates long-term temporal information (i.e. date and time) with short-term pose observations as input. A sequence-to-sequence LSTM encoder-decoder is trained, which encodes observations into LSTM and then decodes as predictions. For deployment, it can perform on-the-fly prediction in real-time. Instead of using manually annotated data, we rely on a robust human detection, tracking and SLAM system, providing us with examples in a global coordinate system. We validate the approach using more than 15K pedestrian trajectories recorded in a care home environment over a period of three months. The experiment shows that the proposed T-Pose-LSTM model advances the state-of-the-art 2D-based method for human trajectory prediction in long-term mobile robot deployments.
---
paper_title: A mobile robot that understands pedestrian spatial behaviors

paper_content:

In human society, there are many invisible social rules or spatial effects existing in our environments. The robot that does not comprehend these spatial effects might harm people or itself. This paper presents a spatial behavior cognition model (SBCM) to describe the spatial effects existing between people and people, people and environments. By understanding the spatial effects in human-lived environments, the robot not only predicts pedestrian intentions and trajectories but also behaves socially acceptable motions. Moreover, the concept of pedestrian ego-graph (PEG) is proposed to efficiently query pedestrian-like paths for trajectory prediction. Model evaluation and experiments are shown to verify the proposed idea in this paper.
---
paper_title: Human Motion Prediction Under Social Grouping Constraints

paper_content:

Accurate long-term prediction of human motion in populated spaces is an important but difficult task for mobile robots and intelligent vehicles. What makes this task challenging is that human motion is influenced by a large variety of factors including the person's intention, the presence, attributes, actions, social relations and social norms of other surrounding agents, and the geometry and semantics of the environment. In this paper, we consider the problem of computing human motion predictions that account for such factors. We formulate the task as an MDP planning problem with stochastic policies and propose a weighted random walk algorithm in which each agent is locally influenced by social forces from other nearby agents. The novelty of this paper is that we incorporate social grouping information into the prediction process reflecting the soft formation constraints that groups typically impose to their members' motion. We show that our method makes more accurate predictions than three state-of-the-art methods in terms of probabilistic and geometrical performance metrics.
---
paper_title: Unsupervised robot learning to predict person motion

paper_content:

Socially interacting robots will need to understand the intentions and recognize the behaviors of people they come in contact with. In this paper we look at how a robot can learn to recognize and predict people's intended path based on its own observations of people over time. Our approach uses people tracking on the robot from either RGBD cameras or LIDAR. The tracks are separated into homogeneous motion classes using a pre-trained SVM. Then the individual classes are clustered and prototypes are extracted from each cluster. These are then used to predict a person's future motion based on matching to a partial prototype and using the rest of the prototype as the predicted motion. Results from experiments in a kitchen environment in our lab demonstrate the capabilities of the proposed method.
---
paper_title: Intent-aware long-term prediction of pedestrian motion

paper_content:

We present a method to predict long-term motion of pedestrians, modeling their behavior as jump-Markov processes with their goal a hidden variable. Assuming approximately rational behavior, and incorporating environmental constraints and biases, including time-varying ones imposed by traffic lights, we model intent as a policy in a Markov decision process framework. We infer pedestrian state using a Rao-Blackwellized filter, and intent by planning according to a stochastic policy, reflecting individual preferences in aiming at the same goal.
---
paper_title: Multimodal Interaction-aware Motion Prediction for Autonomous Street Crossing

paper_content:

For mobile robots navigating on sidewalks, it is essential to be able to safely cross street intersections. Most existing approaches rely on the recognition of the traffic light signal to make an informed crossing decision. Although these approaches have been crucial enablers for urban navigation, the capabilities of robots employing such approaches are still limited to navigating only on streets containing signalized intersections. In this paper, we address this challenge and propose a multimodal convolutional neural network framework to predict the safety of a street intersection for crossing. Our architecture consists of two subnetworks; an interaction-aware trajectory estimation stream IA-TCNN, that predicts the future states of all observed traffic participants in the scene, and a traffic light recognition stream AtteNet. Our IA-TCNN utilizes dilated causal convolutions to model the behavior of the observable dynamic agents in the scene without explicitly assigning priorities to the interactions among them. While AtteNet utilizes Squeeze-Excitation blocks to learn a content-aware mechanism for selecting the relevant features from the data, thereby improving the noise robustness. Learned representations from the traffic light recognition stream are fused with the estimated trajectories from the motion prediction stream to learn the crossing decision. Furthermore, we extend our previously introduced Freiburg Street Crossing dataset with sequences captured at different types of intersections, demonstrating complex interactions among the traffic participants. Extensive experimental evaluations on public benchmark datasets and our proposed dataset demonstrate that our network achieves state-of-the-art performance for each of the subtasks, as well as for the crossing safety prediction.
---
paper_title: Forecasting Interactive Dynamics of Pedestrians with Fictitious Play

paper_content:

We develop predictive models of pedestrian dynamics by encoding the coupled nature of multi-pedestrian interaction using game theory, and deep learning-based visual analysis to estimate person-specific behavior parameters. Building predictive models for multi-pedestrian interactions however, is very challenging due to two reasons: (1) the dynamics of interaction are complex interdependent processes, where the predicted behavior of one pedestrian can affect the actions taken by others and (2) dynamics are variable depending on an individuals physical characteristics (e.g., an older person may walk slowly while the younger person may walk faster). To address these challenges, we (1) utilize concepts from game theory to model the interdependent decision making process of multiple pedestrians and (2) use visual classifiers to learn a mapping from pedestrian appearance to behavior parameters. We evaluate our proposed model on several public multiple pedestrian interaction video datasets. Results show that our strategic planning model explains human interactions 25% better when compared to state-of-the-art methods.
---
paper_title: Bayesian intention inference for trajectory prediction with an unknown goal destination

paper_content:

Contextual cues can provide a rich source of information for robots that operate in the presence of other agents such as people, animals, vehicles and fellow robots. We are interested in context, in the form of the behavioural intent of an agent, for enhanced trajectory prediction. We present a Bayesian framework that estimates both the intended goal destination and future trajectory of a mobile agent moving among multiple static obstacles. Our method is based on multi-modal hypotheses of the intended goal, and is focused primarily on the long-term trajectory of the agent. We propose a computationally efficient solution and demonstrate its behaviour in a pedestrian scenario with a real-world data set. Results show the benefits of our method in comparison to traditional trajectory prediction methods and illustrate the feasibility of integration with higher-level planning algorithms.
---
paper_title: Who are you with and where are you going?

paper_content:

We propose an agent-based behavioral model of pedestrians to improve tracking performance in realistic scenarios. In this model, we view pedestrians as decision-making agents who consider a plethora of personal, social, and environmental factors to decide where to go next. We formulate prediction of pedestrian behavior as an energy minimization on this model. Two of our main contributions are simple, yet effective estimates of pedestrian destination and social relationships (groups). Our final contribution is to incorporate these hidden properties into an energy formulation that results in accurate behavioral prediction. We evaluate both our estimates of destination and grouping, as well as our accuracy at prediction and tracking against state of the art behavioral model and show improvements, especially in the challenging observational situation of infrequent appearance observations–something that might occur in thousands of webcams available on the Internet.
---
paper_title: Long-term Path Prediction in Urban Scenarios using Circular Distributions

paper_content:

Abstract Human ability to foresee the near future plays a key role in everyone's life to prevent potentially dangerous situations. To be able to make predictions is crucial when people have to interact with the surrounding environment. Modeling such capability can lead to the design of automated warning systems and provide moving robots with an intelligent way of interaction with changing situation. In this work we focus on a typical urban human-scene where we aim at predicting an agent's behavior using a stochastic model. In this approach, we fuse the various factors that would contribute to a human motion in different contexts. Our method uses previously observed trajectories to build point-wise circular distributions that after combination, provide a statistical smooth prediction towards the most likely areas. More specifically, a ray-launching procedure, based on a semantic segmentation, gives a coarse scene representation for collision avoidance; a nearly-constant velocity dynamic model smooths the acceleration progression and knowledge of the agent's destination may further steer the path prediction. Experimental results in structured scenes, validate the effectiveness of the method in predicting paths in comparison to actual trajectories.
---
paper_title: Modeling cooperative navigation in dense human crowds

paper_content:

For robots to be a part of our daily life, they need to be able to navigate among crowds not only safely but also in a socially compliant fashion. This is a challenging problem because humans tend to navigate by implicitly cooperating with one another to avoid collisions, while heading toward their respective destinations. Previous approaches have used handcrafted functions based on proximity to model human-human and human-robot interactions. However, these approaches can only model simple interactions and fail to generalize for complex crowded settings. In this paper, we develop an approach that models the joint distribution over future trajectories of all interacting agents in the crowd, through a local interaction model that we train using real human trajectory data. The interaction model infers the velocity of each agent based on the spatial orientation of other agents in his vicinity. During prediction, our approach infers the goal of the agent from its past trajectory and uses the learned model to predict its future trajectory. We demonstrate the performance of our method against a state-of-the-art approach on a public dataset and show that our model outperforms when predicting future trajectories for longer horizons.
---
paper_title: Intention-aware online POMDP planning for autonomous driving in a crowd

paper_content:

This paper presents an intention-aware online planning approach for autonomous driving amid many pedestrians. To drive near pedestrians safely, efficiently, and smoothly, autonomous vehicles must estimate unknown pedestrian intentions and hedge against the uncertainty in intention estimates in order to choose actions that are effective and robust. A key feature of our approach is to use the partially observable Markov decision process (POMDP) for systematic, robust decision making under uncertainty. Although there are concerns about the potentially high computational complexity of POMDP planning, experiments show that our POMDP-based planner runs in near real time, at 3 Hz, on a robot golf cart in a complex, dynamic environment. This indicates that POMDP planning is improving fast in computational efficiency and becoming increasingly practical as a tool for robot planning under uncertainty. © 2015 IEEE
---
paper_title: Pedestrian Prediction by Planning Using Deep Neural Networks

paper_content:

Accurate traffic participant prediction is the prerequisite for collision avoidance of autonomous vehicles. In this work, we propose to predict pedestrians using goal-directed planning. For this, we infer a mixture density function for possible destinations. We use these destinations as the goal states of a planning stage that performs motion prediction based on common behavior patterns. The patterns are learned by a fully convolutional network operating on maps of the environment. We show that this entire system can be modeled as one monolithic neural network and trained via inverse reinforcement learning. Experimental validation on real world data shows the system's ability to predict both, destinations and trajectories accurately.
---
paper_title: The Walking Behaviour of Pedestrian Social Groups and Its Impact on Crowd Dynamics

paper_content:

Human crowd motion is mainly driven by self-organized processes based on local interactions among pedestrians. While most studies of crowd behaviour consider only interactions among isolated individuals, it turns out that up to 70% of people in a crowd are actually moving in groups, such as friends, couples, or families walking together. These groups constitute medium-scale aggregated structures and their impact on crowd dynamics is still largely unknown. In this work, we analyze the motion of approximately 1500 pedestrian groups under natural condition, and show that social interactions among group members generate typical group walking patterns that influence crowd dynamics. At low density, group members tend to walk side by side, forming a line perpendicular to the walking direction. As the density increases, however, the linear walking formation is bent forward, turning it into a V-like pattern. These spatial patterns can be well described by a model based on social communication between group members. We show that the V-like walking pattern facilitates social interactions within the group, but reduces the flow because of its "non-aerodynamic" shape. Therefore, when crowd density increases, the group organization results from a trade-off between walking faster and facilitating social exchange. These insights demonstrate that crowd dynamics is not only determined by physical constraints induced by other pedestrians and the environment, but also significantly by communicative, social interactions among individuals.
---
paper_title: Human Motion Behaviour Aware Planner (HMBAP) for path planning in dynamic human environments

paper_content:

For a robot navigating in a human inhabited dynamic environment, the knowledge of how the robot's movement can influence the trajectory of people around it can be very valuable. In this work we present a Human Motion Behaviour Aware Planner (HMBAP) which incorporates a Human Motion Behaviour Model (HMBM) in its planning stage to take advantage of this. HMBM is an obstacle avoidance model for people based on social forces which gives the robot an understanding of how people would react to its planned path. This information is useful for the robot to avoid imminent collisions with people in constricted spaces. The resulting robot behaviour is similar to how a human would move (in terms of avoidance behaviour) in a similar situation. We believe that this is a desirable feature for a robot navigating in a human inhabited environment. Our method shows good human-like navigation behaviour in situations where past methods fail to find a solution.
---
paper_title: Multimodal Probabilistic Model-Based Planning for Human-Robot Interaction

paper_content:

This paper presents a method for constructing human-robot interaction policies in settings where multimodality, i.e., the possibility of multiple highly distinct futures, plays a critical role in decision making. We are motivated in this work by the example of traffic weaving, e.g., at highway on-ramps/off-ramps, where entering and exiting cars must swap lanes in a short distance-a challenging negotiation even for experienced drivers due to the inherent multimodal uncertainty of who will pass whom. Our approach is to learn multimodal probability distributions over future human actions from a dataset of human-human exemplars and perform real-time robot policy construction in the resulting environment model through massively parallel sampling of human responses to candidate robot action sequences. Direct learning of these distributions is made possible by recent advances in the theory of conditional variational autoencoders (CVAEs), whereby we learn action distributions simultaneously conditioned on the present interaction history, as well as candidate future robot actions in order to take into account response dynamics. We demonstrate the efficacy of this approach with a human-in-the-loop simulation of a traffic weaving scenario.
---
paper_title: Walking Ahead: The Headed Social Force Model

paper_content:

Human motion models are finding an increasing number of novel applications in many different fields, such as building design, computer graphics and robot motion planning. The Social Force Model is one of the most popular alternatives to describe the motion of pedestrians. By resorting to a physical analogy, individuals are assimilated to point-wise particles subject to social forces which drive their dynamics. Such a model implicitly assumes that humans move isotropically. On the contrary, empirical evidence shows that people do have a preferred direction of motion, walking forward most of the time. Lateral motions are observed only in specific circumstances, such as when navigating in overcrowded environments or avoiding unexpected obstacles. In this paper, the Headed Social Force Model is introduced in order to improve the realism of the trajectories generated by the classical Social Force Model. The key feature of the proposed approach is the inclusion of the pedestrians' heading into the dynamic model used to describe the motion of each individual. The force and torque representing the model inputs are computed as suitable functions of the force terms resulting from the traditional Social Force Model. Moreover, a new force contribution is introduced in order to model the behavior of people walking together as a single group. The proposed model features high versatility, being able to reproduce both the unicycle-like trajectories typical of people moving in open spaces and the point-wise motion patterns occurring in high density scenarios. Extensive numerical simulations show an increased regularity of the resulting trajectories and confirm a general improvement of the model realism.
---
paper_title: Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning

paper_content:

Finding feasible, collision-free paths for multiagent systems can be challenging, particularly in non-communicating scenarios where each agent's intent (e.g. goal) is unobservable to the others. In particular, finding time efficient paths often requires anticipating interaction with neighboring agents, the process of which can be computationally prohibitive. This work presents a decentralized multiagent collision avoidance algorithm based on a novel application of deep reinforcement learning, which effectively offloads the online computation (for predicting interaction patterns) to an offline learning procedure. Specifically, the proposed approach develops a value network that encodes the estimated time to the goal given an agent's joint configuration (positions and velocities) with its neighbors. Use of the value network not only admits efficient (i.e., real-time implementable) queries for finding a collision-free velocity vector, but also considers the uncertainty in the other agents' motion. Simulation results show more than 26 percent improvement in paths quality (i.e., time to reach the goal) when compared with optimal reciprocal collision avoidance (ORCA), a state-of-the-art collision avoidance strategy.
---
paper_title: Novel planning-based algorithms for human motion prediction

paper_content:

Human motion prediction from visual tracking is a challenging problem with a wide array of applications such as robotics, video surveillance and situation understanding. Recently, planning-based methods -which assume that people move by planning over a cost function- have emerged as one of the most promising alternatives. Nevertheless, state of the art planning based algorithms have shortcomings regarding their computational complexity and ability to predict for arbitrary time intervals. This paper addresses these shortcomings by leveraging alternative planning techniques (Fast Marching Method) and formulating efficient algorithms for goal estimation and full spatiotemporal prediction with lower complexity than comparable approaches. In preliminary experiments, the proposed method significantly outperforms the accuracy of the current state-of-the-art approach while reducing the computation time by a factor of 30 using a parallel version of our algorithm.
---
paper_title: Social LSTM: Human Trajectory Prediction in Crowded Spaces

paper_content:

Pedestrians follow different trajectories to avoid obstacles and accommodate fellow pedestrians. Any autonomous vehicle navigating such a scene should be able to foresee the future positions of pedestrians and accordingly adjust its path to avoid collisions. This problem of trajectory prediction can be viewed as a sequence generation task, where we are interested in predicting the future trajectory of people based on their past positions. Following the recent success of Recurrent Neural Network (RNN) models for sequence prediction tasks, we propose an LSTM model which can learn general human movement and predict their future trajectories. This is in contrast to traditional approaches which use hand-crafted functions such as Social forces. We demonstrate the performance of our method on several public datasets. Our model outperforms state-of-the-art methods on some of these datasets. We also analyze the trajectories predicted by our model to demonstrate the motion behaviour learned by our model.
---
paper_title: Social Attention: Modeling Attention in Human Crowds

paper_content:

Robots that navigate through human crowds need to be able to plan safe, efficient, and human predictable trajectories. This is a particularly challenging problem as it requires the robot to predict future human trajectories within a crowd where everyone implicitly cooperates with each other to avoid collisions. Previous approaches to human trajectory prediction have modeled the interactions between humans as a function of proximity. However, that is not necessarily true as some people in our immediate vicinity moving in the same direction might not be as important as other people that are further away, but that might collide with us in the future. In this work, we propose Social Attention, a novel trajectory prediction model that captures the relative importance of each person when navigating in the crowd, irrespective of their proximity. We demonstrate the performance of our method against a state-of-the-art approach on two publicly available crowd datasets and analyze the trained attention model to gain a better understanding of which surrounding agents humans attend to, when navigating in a crowd.
---
paper_title: Activity forecasting

paper_content:

We address the task of inferring the future actions of people from noisy visual input. We denote this task activity forecasting. To achieve accurate activity forecasting, our approach models the effect of the physical environment on the choice of human actions. This is accomplished by the use of state-of-the-art semantic scene understanding combined with ideas from optimal control theory. Our unified model also integrates several other key elements of activity analysis, namely, destination forecasting, sequence smoothing and transfer learning. As proof-of-concept, we focus on the domain of trajectory-based activity analysis from visual input. Experimental results demonstrate that our model accurately predicts distributions over future actions of individuals. We show how the same techniques can improve the results of tracking algorithms by leveraging information about likely goals and trajectories.
---
paper_title: You'll never walk alone: Modeling social behavior for multi-target tracking

paper_content:

Object tracking typically relies on a dynamic model to predict the object's location from its past trajectory. In crowded scenarios a strong dynamic model is particularly important, because more accurate predictions allow for smaller search regions, which greatly simplifies data association. Traditional dynamic models predict the location for each target solely based on its own history, without taking into account the remaining scene objects. Collisions are resolved only when they happen. Such an approach ignores important aspects of human behavior: people are driven by their future destination, take into account their environment, anticipate collisions, and adjust their trajectories at an early stage in order to avoid them. In this work, we introduce a model of dynamic social behavior, inspired by models developed for crowd simulation. The model is trained with videos recorded from birds-eye view at busy locations, and applied as a motion model for multi-people tracking from a vehicle-mounted camera. Experiments on real sequences show that accounting for social interactions and scene knowledge improves tracking performance, especially during occlusions.
---
paper_title: Crowds by Example

paper_content:

We present an example-based crowd simulation technique. Most crowd simulation techniques assume that the behavior exhibited by each person in the crowd can be defined by a restricted set of rules. This assumption limits the behavioral complexity of the simulated agents. By learning from real-world examples, our autonomous agents display complex natural behaviors that are often missing in crowd simulations. Examples are created from tracked video segments of real pedestrian crowds. During a simulation, autonomous agents search for examples that closely match the situation that they are facing. Trajectories taken by real people in similar situations, are copied to the simulated agents, resulting in seemingly natural behaviors.
---
paper_title: Online learning for human classification in 3D LiDAR-based tracking

paper_content:

Human detection and tracking are essential aspects to be considered in service robotics, as the robot often shares its workspace and interacts closely with humans. This paper presents an online learning framework for human classification in 3D LiDAR scans, taking advantage of robust multi-target tracking to avoid the need for data annotation by a human expert. The system learns iteratively by retraining a classifier online with the samples collected by the robot over time. A novel aspect of our approach is that errors in training data can be corrected using the information provided by the 3D LiDAR-based tracking. In order to do this, an efficient 3D cluster detector of potential human targets has been implemented. We evaluate the framework using a new 3D LiDAR dataset of people moving in a large indoor public space, which is made available to the research community. The experiments analyse the real-time performance of the cluster detector and show that our online learned human classifier matches and in some cases outperforms its offline version.
---
paper_title: Pedestrian Path Prediction with Recursive Bayesian Filters: A Comparative Study

paper_content:

In the context of intelligent vehicles, we perform a comparative study on recursive Bayesian filters for pedestrian path prediction at short time horizons (< 2s). We consider Extended Kalman Filters (EKF) based on single dynamical models and Interacting Multiple Models (IMM) combining several such basic models (constant velocity/acceleration/turn). These are applied to four typical pedestrian motion types (crossing, stopping, bending in, starting). Position measurements are provided by an external state-of-the-art stereo vision-based pedestrian detector. We investigate the accuracy of position estimation and path prediction, and the benefit of the IMMs vs. the simpler single dynamical models. Special care is given to the proper sensor modeling and parameter optimization. The dataset and evaluation framework are made public to facilitate benchmarking.
---
paper_title: Person Tracking in Large Public Spaces Using 3-D Range Sensors

paper_content:

A method for tracking the position, orientation, and height of persons in large public environments is presented. Such a piece of information is known to be useful both for understanding their actions, as well as for applications such as human-robot interaction. We use multiple 3-D range sensors, which are mounted above human height to have less occlusion between persons. A computationally simple-tracking method is proposed that works on single sensor data and combines multiple sensors so that large areas can be covered with a minimum number of sensors. Moreover, it can work with different sensor types and is robust to the imperfect sensor measurements; therefore, it is possible to combine currently available 3-D range sensor solutions to achieve tracking in wide public spaces. The method was implemented in a shopping center environment, and it was shown that good tracking performance can be achieved.
---
paper_title: Understanding collective crowd behaviors: Learning a Mixture model of Dynamic pedestrian-Agents

paper_content:

In this paper, a new Mixture model of Dynamic pedestrian-Agents (MDA) is proposed to learn the collective behavior patterns of pedestrians in crowded scenes. Collective behaviors characterize the intrinsic dynamics of the crowd. From the agent-based modeling, each pedestrian in the crowd is driven by a dynamic pedestrian-agent, which is a linear dynamic system with its initial and termination states reflecting a pedestrian's belief of the starting point and the destination. Then the whole crowd is modeled as a mixture of dynamic pedestrian-agents. Once the model is unsupervisedly learned from real data, MDA can simulate the crowd behaviors. Furthermore, MDA can well infer the past behaviors and predict the future behaviors of pedestrians given their trajectories only partially observed, and classify different pedestrian behaviors in the scene. The effectiveness of MDA and its applications are demonstrated by qualitative and quantitative experiments on the video surveillance dataset collected from the New York Grand Central Station.
---
paper_title: Stable multi-target tracking in real-time surveillance video

paper_content:

The majority of existing pedestrian trackers concentrate on maintaining the identities of targets, however systems for remote biometric analysis or activity recognition in surveillance video often require stable bounding-boxes around pedestrians rather than approximate locations. We present a multi-target tracking system that is designed specifically for the provision of stable and accurate head location estimates. By performing data association over a sliding window of frames, we are able to correct many data association errors and fill in gaps where observations are missed. The approach is multi-threaded and combines asynchronous HOG detections with simultaneous KLT tracking and Markov-Chain Monte-Carlo Data Association (MCM-CDA) to provide guaranteed real-time tracking in high definition video. Where previous approaches have used ad-hoc models for data association, we use a more principled approach based on a Minimal Description Length (MDL) objective which accurately models the affinity between observations. We demonstrate by qualitative and quantitative evaluation that the system is capable of providing precise location estimates for large crowds of pedestrians in real-time. To facilitate future performance comparisons, we make a new dataset with hand annotated ground truth head locations publicly available.
---
paper_title: The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems

paper_content:

Scenario-based testing for the safety validation of highly automated vehicles is a promising approach that is being examined in research and industry. This approach heavily relies on data from real-world scenarios to derive the necessary scenario information for testing. Measurement data should be collected at a reasonable effort, contain naturalistic behavior of road users and include all data relevant for a description of the identified scenarios in sufficient quality. However, the current measurement methods fail to meet at least one of the requirements. Thus, we propose a novel method to measure data from an aerial perspective for scenario-based validation fulfilling the mentioned requirements. Furthermore, we provide a large-scale naturalistic vehicle trajectory dataset from German highways called highD. We evaluate the data in terms of quantity, variety and contained scenarios. Our dataset consists of 16.5 hours of measurements from six locations with 110 000 vehicles, a total driven distance of 45 000 km and 5600 recorded complete lane changes. The highD dataset is available online at: http://www.highD-dataset.com
---
paper_title: You'll never walk alone: Modeling social behavior for multi-target tracking

paper_content:

Object tracking typically relies on a dynamic model to predict the object's location from its past trajectory. In crowded scenarios a strong dynamic model is particularly important, because more accurate predictions allow for smaller search regions, which greatly simplifies data association. Traditional dynamic models predict the location for each target solely based on its own history, without taking into account the remaining scene objects. Collisions are resolved only when they happen. Such an approach ignores important aspects of human behavior: people are driven by their future destination, take into account their environment, anticipate collisions, and adjust their trajectories at an early stage in order to avoid them. In this work, we introduce a model of dynamic social behavior, inspired by models developed for crowd simulation. The model is trained with videos recorded from birds-eye view at busy locations, and applied as a motion model for multi-people tracking from a vehicle-mounted camera. Experiments on real sequences show that accounting for social interactions and scene knowledge improves tracking performance, especially during occlusions.
---
paper_title: Group and Single Pedestrian Behavior in Crowd Dynamics

paper_content:

An analytical study was proposed in this paper to explore various behavioral aspects of group and single pedestrians. The results were obtained from reviewing the literature and studying primary data. Data collected from field surveys, direct observations, and video recordings were analyzed to produce the following findings. For crowds with densities between 0.4 and 0.8 persons per square meter, individuals walked faster than groups, on average. There was also significant evidence to suggest that individuals were likely to have more trajectory changes than groups. As the crowd density got higher, the likelihood of pedestrians splitting from groups of two decreased. With higher crowd densities, the rate at which split members came back to the group decreased. More individual pedestrians chose spacious stairs; groups of two were more likely to go for escalators.
---
paper_title: Autonomous Vehicles That Interact With Pedestrians: A Survey of Theory and Practice

paper_content:

One of the major challenges that autonomous cars are facing today is driving in urban environments. To make it a reality, autonomous vehicles require the ability to communicate with other road users and understand their intentions. Such interactions are essential between vehicles and pedestrians, the most vulnerable road users. Understanding pedestrian behavior, however, is not intuitive and depends on various factors, such as demographics of the pedestrians, traffic dynamics, environmental conditions, and so on. In this paper, we identify these factors by surveying pedestrian behavior studies, both the classical works on pedestrian-driver interaction and the modern ones that involve autonomous vehicles. To this end, we will discuss various methods of studying pedestrian behavior and analyze how the factors identified in the literature are interrelated. We will also review the practical applications aimed at solving the interaction problem, including design approaches for autonomous vehicles that communicate with pedestrians and visual perception and reasoning algorithms tailored to understanding pedestrian intention. Based on our findings, we will discuss the open problems and propose future research directions.
---
paper_title: Forecasting Interactive Dynamics of Pedestrians with Fictitious Play

paper_content:

We develop predictive models of pedestrian dynamics by encoding the coupled nature of multi-pedestrian interaction using game theory, and deep learning-based visual analysis to estimate person-specific behavior parameters. Building predictive models for multi-pedestrian interactions however, is very challenging due to two reasons: (1) the dynamics of interaction are complex interdependent processes, where the predicted behavior of one pedestrian can affect the actions taken by others and (2) dynamics are variable depending on an individuals physical characteristics (e.g., an older person may walk slowly while the younger person may walk faster). To address these challenges, we (1) utilize concepts from game theory to model the interdependent decision making process of multiple pedestrians and (2) use visual classifiers to learn a mapping from pedestrian appearance to behavior parameters. We evaluate our proposed model on several public multiple pedestrian interaction video datasets. Results show that our strategic planning model explains human interactions 25% better when compared to state-of-the-art methods.
---
paper_title: A Game-Theoretic Approach to Replanning-Aware Interactive Scene Prediction and Planning

paper_content:

This paper presents a novel cooperative-driving prediction and planning framework for dynamic environments based on the methods of game theory. The proposed algorithm can be used for highly automated driving on highways or as a sophisticated prediction module for advanced driver-assistance systems with no need for intervehicle communication. The main contribution of this paper is a model-based interaction-aware motion prediction of all vehicles in a scene. In contrast to other state-of-the-art approaches, the system also models the replanning capabilities of all drivers. With that, the driving strategy is able to capture complex interactions between vehicles, thus planning maneuver sequences over longer time horizons. It also enables an accurate prediction of traffic for the next immediate time step. The prediction model is supported by an interpretation of what other drivers intend to do, how they interact with traffic, and the ongoing observation. As part of the prediction loop, the proposed planning strategy incorporates the expected reactions of all traffic participants, offering cooperative and robust driving decisions. By means of experimental results under simulated highway scenarios, the validity of the proposed concept and its real-time capability is demonstrated.
---
paper_title: Age and Group-driven Pedestrian Behaviour: from Observations to Simulations

paper_content:

The development of pedestrian simulation systems requires the acquisition of empirical evidences about human behaviour for sake of model validation. In this framework, the paper presents the results of an on field observation of pedestrian behaviour in an urban crowded walkway. The research was aimed at testing the potentially combined effect of ageing and grouping on speed and proxemic behaviour. In particular, we focused on dyads, as the most frequent type of groups in the observed scenario. Results showed that in situation of irregular flows elderly pedestrians walked the 40% slower than adults, due to locomotion skill decline. Dyads walked the 30% slower than singles, due to the need to maintain spatial cohesion to communicate (proxemics). Results contributed to refine the parametric validation of the agent-based simulation system ELIAS38.
---
paper_title: An Optimality Principle Governing Human Walking

paper_content:

In this paper, we investigate different possible strategies underlying the formation of human locomotor trajectories in goal-directed walking. Seven subjects were asked to walk within a motion capture facility from a fixed starting point and direction, and to cross over distant porches for which both position and direction in the room were changed over trials. Stereotyped trajectories were observed in the different subjects. The underlying idea to attack this question has been to relate this problem to an optimal control scheme: the trajectory is chosen according to some optimization principle. This is our basic starting assumption. The subject being viewed as a controlled system, we tried to identify several criteria that could be optimized. Is it the time to perform the trajectory? The length of the path? The minimum jerk along the path? We found that the variation (time derivative) of the curvature of the locomotor paths is minimized. Moreover, we show that the human locomotor trajectories are well approximated by the geodesics of a differential system minimizing the norm of the control. Such geodesics are made of arcs of clothoids. The clothoid or Cornu spiral is a curve, whose curvature grows with the distance from the origin.
---

Title: Human Motion Trajectory Prediction: A Survey

Section 1: Introduction
Description 1: Provide an overview of the importance and scope of human motion trajectory prediction, its challenges, and the structure of the survey.

Section 2: Overview and Terminology
Description 2: Explain the basic elements of motion prediction, key terminologies, and foundational concepts relevant to the field.

Section 3: Application Domains
Description 3: Discuss the various application domains of motion prediction, such as service robots, self-driving vehicles, and surveillance systems.

Section 4: Related Surveys
Description 4: Review other surveys from different scientific communities that address human motion prediction and related areas.

Section 5: Taxonomy
Description 5: Introduce the proposed taxonomy for motion prediction, detailing the categories based on the modeling approach and the type of contextual cues.

Section 6: Modeling Approach
Description 6: Describe the different modeling approaches for motion prediction, including physics-based, pattern-based, and planning-based methods.

Section 7: Contextual Cues
Description 7: Categorize and explain the contextual cues that influence motion behavior, focusing on cues related to the target agent, other dynamic agents, and the static environment.

Section 8: Classification Rules
Description 8: Discuss the criteria and rules used to classify different methods within the proposed taxonomy.

Section 9: Physics-based Approaches
Description 9: Detail the approaches that use explicit dynamical models based on physics to predict human motion, including single and multi-model methods.

Section 10: Pattern-based Approaches
Description 10: Explain methods that learn motion patterns from data to predict trajectories, including sequential and non-sequential models.

Section 11: Planning-based Approaches
Description 11: Introduce methods that reason about agents' goals and compute policies or paths to achieve these goals, covering forward and inverse planning approaches.

Section 12: Contextual Cues
Description 12: Overview the classes of contextual cues, focusing on the target agent, dynamic environment, and static environment.

Section 13: Motion Prediction Evaluation
Description 13: Review common metrics and datasets used to evaluate motion prediction performance.

Section 14: Discussion
Description 14: Summarize and discuss the state of the art, posing questions regarding the maturity and limitations of current prediction methods.

Section 15: Future Directions
Description 15: Suggest potential future research directions and areas for improvement in human motion trajectory prediction.

Section 16: Conclusions
Description 16: Provide concluding remarks and summarize the key findings from the survey.