A Survey of Computer Vision-Based Human Motion Capture
---
paper_title: The Visual Analysis of Human Movement: A Survey

paper_content:

The ability to recognize humans and their activities by vision is key for a machine to interact intelligently and effortlessly with a human-inhabited environment. Because of many potentially important applications, “looking at people” is currently one of the most active application domains in computer vision. This survey identifies a number of promising applications and provides an overview of recent developments in this domain. The scope of this survey is limited to work on whole-body or hand motion; it does not include work on human faces. The emphasis is on discussing the various methodologies; they are grouped in 2-D approaches with or without explicit shape models and 3-D approaches. Where appropriate, systems are reviewed. We conclude with some thoughts about future directions.
---
paper_title: Motion-based recognition a survey

paper_content:

Abstract Motion-based recognition deals with the recognition of an object or its motion based on motion in a sequence of images. In this approach, a sequence containing a large number of frames is used to extract motion information. The advantage is that a longer sequence leads to recognition of higher level motions, like walking or running, which consists of a complex and coordinated series of events that cannot be understood by looking at only a few frames. This paper provides a review of recent developments in the computer vision aspect of motionbased recognition. We will identify two main steps in motionbased recognition. The first step is the extraction of motion information and its organization into motion models. The second step consists of the matching of some unknown input with a constructed model. Several methods for the recognition of objects and motions will then be reported. They include methods such as cyclic motion detection and recognition, lipreading, hand gestures interpretation, motion verb recognition and temporal textures classification. Tracking and recognition of human motion, like walking, skipping and running will also be discussed. Finally, we will conclude the paper with some thoughts about future directions for motionbased recognition.
---
paper_title: Human motion analysis: A review

paper_content:

Human motion analysis is receiving increasing attention from computer vision researchers. This interest is motivated by a wide spectrum of applications, such as athletic performance analysis, surveillance, man?machine interfaces, content-based image storage and retrieval, and video conferencing. This paper gives an overview of the various tasks involved in motion analysis of the human body. We focus on three major areas related to interpreting human motion: (1) motion analysis involving human body parts, (2) tracking a moving human from a single view or multiple camera perspectives, and (3) recognizing human activities from image sequences. Motion analysis of human body parts involves the low-level segmentation of the human body into segments connected by joints and recovers the 3D structure of the human body using its 2D projections over a sequence of images. Tracking human motion from a single view or multiple perspectives focuses on higher-level processing, in which moving humans are tracked without identifying their body parts. After successfully matching the moving human image from one frame to another in an image sequence, understanding the human movements or activities comes naturally, which leads to our discussion of recognizing human activities.
---
paper_title: Visual surveillance of human activity

paper_content:

W4 is a real time visual surveillance system for detecting and tracking people and monitoring their activities in an outdoor environment. It operates on monocular grayscale video imagery, or on video imagery from an IR camera.Unlike many of systems for tracking people, W4 makes no use of color cues. Instead, W4 employs a combination of shape analysis and tracking to create models of people's appearance so that they can be tracked through interactions such as occlusions. W4 is capable of simultaneously tracking multiple people even with occlusion. It runs at 20 Hz for 320 X 240 resolution images on a dual-pentium 200 PC.© (1998) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.
---
paper_title: Tracking human motion using multiple cameras

paper_content:

Presents a framework for tracking human motion in an indoor environment from sequences of monocular grayscale images obtained from multiple fixed cameras. Multivariate Gaussian models are applied to find the most likely matches of human subjects between consecutive frames taken by cameras mounted in various locations. Experimental results from real data show the robustness of the algorithm and its potential for real time applications.
---
paper_title: Pfinder: real-time tracking of the human body

paper_content:

Pfinder is a real-time system for tracking and interpretation of people. It runs on a standard SGI Indy computer, and has performed reliably on thousands of people in many different physical locations. The system uses a multiclass statistical model of color and shape to segment a person from a background scene, and implements heuristics which can find and track people's head and hands in a wide range of viewing conditions. Pfinder produces a real- time representation of a user useful for applications such as wireless interfaces, video databases, and low-bandwidth coding, without cumbersome wires or attached sensors.© (1996) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.
---
paper_title: Human Movement Analysis Based On Explicit Motion Models

paper_content:

Within the field of computer vision the automatic interpretation of human movements is one of the most challenging tasks. A central problem in analyzing such movements is due to the fact that the human body consists of body parts linked to each other at joints which allows different movements of the parts. Therefore, the human body generally has to be treated as a nonrigid or more precisely as an articulated body. In addition, for general camera positions always some of the body parts are occluded. Although occlusions can provide important cues in a recognition task, the automatic interpretation is more difficult. Another problem that has to be dealt with is the clothing which can have a large influence on the appearence of a person (wide or tight trousers, different jackets, etc.). Clothing can also cause complex illumination phenomena that, in addition, change during movement (compare with efforts in the field of computer graphics to simulate cloth objects, e.g., [83]).
---
paper_title: Whole-body modelling of people from multiview images to populate virtual worlds

paper_content:

In this paper a new technique is introduced for automatically building recognisable, moving 3D models of individual people. A set of multiview colour images of a person is captured from the front, sides and back by one or more cameras. Model-based reconstruction of shape from silhouettes is used to transform a standard 3D generic humanoid model to approximate a person's shape and anatomical structure. Realistic appearance is achieved by colour texture mapping from the multiview images. The results show the reconstruction of a realistic 3D facsimile of the person suitable for animation in a virtual world. The system is inexpensive and is reliable for large variations in shape, size and clothing. This is the first approach to achieve realistic model capture for clothed people and automatic reconstruction of animated models. A commercial system based on this approach has recently been used to capture thousands of models of the general public.
---
paper_title: A system for human motion matching between synthetic and real images based on a biomechanic graphical model

paper_content:

A system for analysis and synthesis of human motion is presented. The system consists of an analysis part and a synthesis part. The analysis part can be used automatically or interactively. The automatic analysis part includes the pre-processing, modeling, matching and interpretation phases. The interactive analysis part includes the same phases but with the possibility of user supervision. We present a global overview of the whole system. The user can define a biomechanic graphical model to represent the human body in a 3D space, and use tools to perform an automatic or interactive supervised matching between the animated model and the real images to recover the motion. The synthesis part uses the results of matching process to show the motion of the human body that is shown in the real images, from any viewpoint. We use specific criteria to match walking persons from different views. Some results and images are presented. >
---
paper_title: An efficient method for contour tracking using active shape models

paper_content:

There has been considerable research interest recently, in the areas of real time contour tracking and active shape models. This paper demonstrates how dynamic filtering can be used in combination with a modal-based flexible shape model to track an articulated non-rigid body in motion. The results show the method being used to track the silhouette of a walking pedestrian in real time. The active shape model used was generated automatically from real image data and incorporates variability in shape due to orientation as well as object flexibility. A Kalman filter is used to control spatial scale for feature search over successive frames. Iterative refinement allows accurate contour localisation where feasible. The shape model incorporates knowledge of the likely shape of the contour and speeds up tracking by reducing the number of system parameters. A further increase in speed is obtained by filtering the shape parameters independently. >
---
paper_title: Constructing virtual worlds using dense stereo

paper_content:

We present Virtualized Reality, a technique to create virtual worlds out of dynamic events using densely distributed stereo views. The intensity image and depth map for each camera view at each time instant are combined to form a Visible Surface Model. Immersive interaction with the virtualized event is possible using a dense collection of such models. Additionally, a Complete Surface Model of each instant can be built by merging the depth maps from different cameras into a common volumetric space. The corresponding model is compatible with traditional virtual models and can be interacted with immersively using standard tools. Because both VSMs and CSMs are fully three-dimensional, virtualized models can also be combined and modified to build larger, more complex environments, an important capability for many non-trivial applications. We present results from 3D Dome, our facility to create virtualized models.
---
paper_title: The convergence of graphics and vision

paper_content:

Computer graphics and computer vision are inverse problems. Traditional computer graphics starts with input geometric models and produces image sequences. Traditional computer vision starts with input image sequences and produces geometric models. Lately, there has been a meeting in the middle, and the center, the prize, is to create stunning images in real time. Vision researchers now work from images backward, just as far backward as necessary to create models that capture a scene without going to full geometric models. Graphics researchers now work with hybrid geometry and image models. Approaching similar problems from opposite directions, graphics and vision researchers are reaching a fertile middle ground. The goal is to find the best possible tools for the imagination. This overview describes cutting edge work, some of which will debut at Siggraph 98.
---
paper_title: Human motion analysis based on a robot arm model

paper_content:

A model-based method for analyzing a human body motion is presented. The method is based on a robot arm model which represents a human body motion. Combining the model and the gradient scheme, the movement of the configuration of the model (Human) can be directly estimated from an image sequence. >
---
paper_title: Towards model-based capture of a persons shape, appearance and motion

paper_content:

This paper introduces a model-based approach to capturing a persons shape, appearance and movement. A 3D animated model of a clothed persons whole-body shape and appearance is automatically constructed from a set of orthogonal view colour images. The reconstructed model of a person is then used together with the least-squares inverse-kinematics framework of Bregler and Malik (1998) to capture simple 3D movements from a video image sequence.
---
paper_title: Tracking and counting moving people

paper_content:

The development of a system able to detect and track moving people is described. Given a sequence of time-varying images, the goal is to count the number of people crossing a counting line. A motion detection module first determines whether any person has entered the scene; a tracking module combining prediction and matching then follows people until they reach the counting line. The efficiency of the proposed approach is illustrated and assessed using sequences of images taken in a real environment. >
---
paper_title: Human body limbs tracking by multi-ocular vision

paper_content:

This article describes a tracking method of human body limbs from multi-ocular sequence of perspective images. These limbs and the associated articulations must be first modelled. During a learning step, we model the texture linked to the limbs. The lack of characteristic points on the skin is compensated by the wearing a non-repetitive texture tight. The principle of the method is based on the interpretation of image textured patterns as the SD perspective projections of points of the textured articulated model. An iterative Levenberg-Marquardt process is used to compute the model pose in accordance with the analysed image. The calculated attitude is filtered to predict the model pose in the following image of the sequence. The image patterns are extracted locally according to the textured articulated model in the predicted attitude. Tracking experiments, illustrated in this paper with cycling sequences, demonstrate the validity of the approach.
---
paper_title: A model based approach in extracting and generating human motion

paper_content:

Introduces a model based approach in acquiring motion of a human body from a video image sequence. A model can not only help recognition of a human in the images, but also save the cost in 3D sensing. A 3D human model which has the function of articulate motion and changeable size, color and surface shape is provided. The model can be driven either automatically or manually through a graphics interface. We first fit the model to some particular frames so as to yield a personal model. The matching of the personal model with the image sequence for extracting human motion starts from matching key frames. The motion between key frames is interpolated. We are building a system for generating motion for multimedia, virtual world, and animation.
---
paper_title: Pfinder: real-time tracking of the human body

paper_content:

Pfinder is a real-time system for tracking and interpretation of people. It runs on a standard SGI Indy computer, and has performed reliably on thousands of people in many different physical locations. The system uses a multiclass statistical model of color and shape to segment a person from a background scene, and implements heuristics which can find and track people's head and hands in a wide range of viewing conditions. Pfinder produces a real- time representation of a user useful for applications such as wireless interfaces, video databases, and low-bandwidth coding, without cumbersome wires or attached sensors.© (1996) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.
---
paper_title: MDL-based spatiotemporal segmentation from motion in a long image sequence

paper_content:

This paper presents a method for spatiotemporal segmentation of long sequences of images which include multiple independently moving objects, based on the Minimum Description Length (MDL) principle. Spatiotemporal (ST) segments in the image sequence are extracted, each of which consists of edge segments having similar motions. First, we construct a family of motion models, each of which is completely determined by its specified set of equations. Then we formulate the motion description length in a long sequence based on these sets of equations. The motion state of an object at a given moment is determined by finding the model with shortest description length. Temporal segmentation is carried out when the motion state is found to have changed. At the same time, the spatial segmentation is globally optimized in such a way that the motion description of the entire scene reaches a minimum. >
---
paper_title: Stereoscopic system for human body tracking in natural scenes

paper_content:

Human body detection and tracking in a scene constitutes a very active working field clue to their applicability to many areas, specially as a man-machine interface (MMI) means. The system presented aims to improve the reliability and efficiency of teleoperation. The system is of application to teleoperated manipulation in civil applications such as big robots in shipyards, mines, public works or even cranes. Image segmentation is performed from movement detection. The recognition of moving bodies is verified by means of a simplified articulated cylindrical model, thus allowing to operate with a low computational cost.
---
paper_title: Human motion analysis based on a robot arm model

paper_content:

A model-based method for analyzing a human body motion is presented. The method is based on a robot arm model which represents a human body motion. Combining the model and the gradient scheme, the movement of the configuration of the model (Human) can be directly estimated from an image sequence. >
---
paper_title: Tracking people with twists and exponential maps

paper_content:

This paper demonstrates a new visual motion estimation technique that is able to recover high degree-of-freedom articulated human body configurations in complex video sequences. We introduce the use of a novel mathematical technique, the product of exponential maps and twist motions, and its integration into a differential motion estimation. This results in solving simple linear systems, and enables us to recover robustly the kinematic degrees-of-freedom in noise and complex self occluded configurations. We demonstrate this on several image sequences of people doing articulated full body movements, and visualize the results in re-animating an artificial 3D human model. We are also able to recover and re-animate the famous movements of Eadweard Muybridge's motion studies from the last century. To the best of our knowledge, this is the first computer vision based system that is able to process such challenging footage and recover complex motions with such high accuracy.
---
paper_title: Learning and recognizing human dynamics in video sequences

paper_content:

This paper describes a probabilistic decomposition of human dynamics at multiple abstractions, and shows how to propagate hypotheses across space, time, and abstraction levels. Recognition in this framework is the succession of very general low level grouping mechanisms to increased specific and learned model based grouping techniques at higher levels. Hard decision thresholds are delayed and resolved by higher level statistical models and temporal context. Low-level primitives are areas of coherent motion found by EM clustering, mid-level categories are simple movements represented by dynamical systems, and high-level complex gestures are represented by Hidden Markov Models as successive phases of ample movements. We show how such a representation can be learned from training data, and apply It to the example of human gait recognition.
---
paper_title: Pfinder: real-time tracking of the human body

paper_content:

Pfinder is a real-time system for tracking and interpretation of people. It runs on a standard SGI Indy computer, and has performed reliably on thousands of people in many different physical locations. The system uses a multiclass statistical model of color and shape to segment a person from a background scene, and implements heuristics which can find and track people's head and hands in a wide range of viewing conditions. Pfinder produces a real- time representation of a user useful for applications such as wireless interfaces, video databases, and low-bandwidth coding, without cumbersome wires or attached sensors.© (1996) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.
---
paper_title: A new robust real-time method for extracting human silhouettes from color images

paper_content:

A new method for extracting human silhouettes from color images in real-time regardless of changes in lighting conditions and backgrounds is proposed. In the method, RGB values of each pixel in color images acquired by a CCD camera are converted to YIQ values. Before extracting human silhouettes, the mean YIQ values of each pixel and the threshold values for the Y, I, Q components are obtained in advance from the image sequence acquired by the camera that observed the background. Based on the mean values and the threshold values, each pixel is judged whether it belongs to the human silhouette or not. The proposed method was used with the virtual metamorphosis system, in which anyone can change his or her form to any other form, and the robustness and real-time-ness were confirmed.
---
paper_title: Vision-based animation of digital humans

paper_content:

The paper presents a system for animating customized virtual humans using motion parameters estimated from multi-view image sequences. The advantage of the method is that the subject does not have to wear markers or other devices. The authors offer a detailed analysis of the singularities that might arise during tracking humans using multiple cameras and provide an analysis of the criteria that allow the ordering of the cameras that provide the most information for tracking. They demonstrate their system using image sequences requiring the recovery of complex three-dimensional motions in the presence of significant occlusion, and present a quantitative analysis of the accuracy of the estimated motion parameters.
---
paper_title: Reach out and touch space (motion learning)

paper_content:

We propose a method for learning models of human motion from a coarsely sampled set of examples. The models we synthesize may be used to generate plausible motions from a high level description consisting of start and stop positions, style, mood, age, etc. In the field of computer vision, such models can be useful for human body motion tracking/estimation and gesture recognition. The models can also be used to generate arbitrary realistic human motion, and may be of help in trying to understand the mechanisms behind the perception of biological motion by the human visual system. Experimental results of the learning technique applied to reaching and drawing motions are presented.
---
paper_title: Recognition of human body motion using phase space constraints

paper_content:

A new method for representing and recognizing human body movements is presented. The basic idea is to identify sets of constraints that are diagnostic of a movement: expressed using body-centered coordinates such as joint angles and in force only during a particular movement. Assuming the availability of Cartesian tracking data, we develop techniques for a representation of movements defined by space curves in subspaces of a "phase space." The phase space has axes of joint angles and torso location and attitude, and the axes of the subspaces are subsets of the axes of the phase space. Using this representation we develop a system for learning new movements from ground truth data by searching for constraints. We then use the learned representation for recognizing movements in unsegmented data. We train and test the system on nine fundamental steps from classical ballet performed by two dancers; the system accurately recognizes the movements in the unsegmented stream of motion. >
---
paper_title: Lower limb kinematics of human walking with the medial axis transformation

paper_content:

The paper describes a simple model for free-speed human walking and compares ordinary images of a walking person to the model. Three dimensional kinematic data were obtained from subjects walking with markers over the joints of their limbs. The average of these data was used to derive a model stick figure of the lower limbs, based on the average anthropometric data of the population. Stick figures were obtained from ordinary images of persons dressed in tight fitting clothes without any markers by using the medial axis transformation. The two dimensional information from the image stick figures was compared with the projection of the three dimensional information of the model onto the relevant plane. A high degree of correlation was noted between the rotational patterns of the model and image stick figures. >
---
paper_title: LOG-TRACKER: AN ATTRIBUTE-BASED APPROACH TO TRACKING HUMAN BODY MOTION

paper_content:

Motion provides extra information that can aid in the recognition of objects. One of the most commonly seen objects is, perhaps, the human body. Yet little attention has been paid to the analysis of human motion. One of the key steps required for a successful motion analysis system is the ability to track moving objects. In this paper, we describe a new system called Log-Tracker, which was recently developed for tracking the motion of the different parts of the human body. Occlusion of body parts is termed a forking condition. Two classes of forks as well as the attributes required to classify them are described. Experimental results from two gymnastics sequences indicate that the system is able to track the body parts even when they are occluded for a short period of time. Occlusions that extend for a long period of time still pose problems to Log-Tracker.
---
paper_title: An efficient method for contour tracking using active shape models

paper_content:

There has been considerable research interest recently, in the areas of real time contour tracking and active shape models. This paper demonstrates how dynamic filtering can be used in combination with a modal-based flexible shape model to track an articulated non-rigid body in motion. The results show the method being used to track the silhouette of a walking pedestrian in real time. The active shape model used was generated automatically from real image data and incorporates variability in shape due to orientation as well as object flexibility. A Kalman filter is used to control spatial scale for feature search over successive frames. Iterative refinement allows accurate contour localisation where feasible. The shape model incorporates knowledge of the likely shape of the contour and speeds up tracking by reducing the number of system parameters. A further increase in speed is obtained by filtering the shape parameters independently. >
---
paper_title: Real-time estimation of human body posture from monocular thermal images

paper_content:

This paper introduces a new real-time method to estimate the posture of a human from thermal images acquired by an infrared camera regardless of the back-ground and lighting conditions. Distance transformation is performed for the human body area extracted from the thresholded thermal image for the. Calculation of the center of gravity. After the orientation of the upper half of the body is obtained by calculating the moment of inertia, significant points such as the top of the head, the tips of the hands and foot are heuristically located. In addition, the elbow and foot positions are estimated from the detected (significant) points using a genetic algorithm based learning procedure. The experimental results demonstrate the robustness of the proposed algorithm and real-time (faster than 20 frames per second) performance.
---
paper_title: Tracking interacting people

paper_content:

A computer vision system for tracking multiple people in relatively unconstrained environments is described. Tracking is performed at three levels of abstraction: regions, people and groups. A novel, adaptive background subtraction method that combines colour and gradient information is used to cope with shadows and unreliable colour cues. People are tracked through mutual occlusions as they form groups and part from one another. Strong use is made of colour information to disambiguate occlusions and to provide qualitative estimates of depth ordering and position during occlusion. Some simple interactions with objects can also be detected. The system is tested using indoor and outdoor sequences. It is robust and should provide a useful mechanism for bootstrapping and reinitialisation of tracking using more-specific but less-robust human models.
---
paper_title: Posture estimation using structure and motion models

paper_content:

Sensing of human motion is very important for human-computer interactive applications such as virtual reality, gesture recognition, and communication. A vision system is suitable for human-computer interaction since this involves passive sensing and the system can estimate the motion of the user without any discomfort for the user. In this paper, we propose an algorithm for fast posture estimation of the user from an image sequence by using the position information of the hands and head. Our algorithm is founded on a model based method. The parameters of a geometric model are calculated from human kinematics. It is inadequate to obtain a unique solution from the image information such as position of the head and hands. The unknown parameters are predicted by the motion models and the previous posture parameters, then the unknown parameters are adjusted by a minimization method.
---
paper_title: Pfinder: real-time tracking of the human body

paper_content:

Pfinder is a real-time system for tracking and interpretation of people. It runs on a standard SGI Indy computer, and has performed reliably on thousands of people in many different physical locations. The system uses a multiclass statistical model of color and shape to segment a person from a background scene, and implements heuristics which can find and track people's head and hands in a wide range of viewing conditions. Pfinder produces a real- time representation of a user useful for applications such as wireless interfaces, video databases, and low-bandwidth coding, without cumbersome wires or attached sensors.© (1996) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.
---
paper_title: Cardboard people: a parameterized model of articulated image motion

paper_content:

We extend the work of Black and Yacoob (1995) on the tracking and recognition of human facial expressions using parametrized models of optical flow to deal with the articulated motion of human limbs. We define a "card-board person model" in which a person's limbs are represented by a set of connected planar patches. The parametrized image motion of these patches in constrained to enforce articulated motion and is solved for directly using a robust estimation technique. The recovered motion parameters provide a rich and concise description of the activity that can be used for recognition. We propose a method for performing view-based recognition of human activities from the optical flow parameters that extends previous methods to cope with the cyclical nature of human motion. We illustrate the method with examples of tracking human legs of long image sequences.
---
paper_title: A model driven 3D image interpretation system applied to person detection in video images

paper_content:

Detecting persons in images is up to now an unsolved problem in general. We propose an approach which strictly follows the assumption that only objects and parts of these objects can be detected, if their appearance is known in advance. We consider the detection step as a first step towards the recognition process. The latter contains the reasoning about the movements of a person. The appearance of a 3D object means the description of the mapping of object features into the image. The proposed system uses a hierarchical model to control the image segmentation and matching process. It is a multipurpose model for articulated objects which is supporting the linkage of different features.
---
paper_title: Toward Non-intrusive Motion Capture

paper_content:

Shape-from-silhouettes, a simple approach to 3D shape understanding, can also be used for recovering posture and motion of human bodies. Silhouettes are easy to obtain from intensity images, do not require foreign objects attached to the subject, can directly provide a 3-D reconstruction of the body, or drive model-based motion capture. The purpose of this work is to demonstrate the effectiveness of this approach in a virtual environment, and to investigate number and position of stationary cameras suitable for precise reconstruction of the 3-D posture and motion of the human body.
---
paper_title: Local and Global Skeleton Fitting Techniques for Optical Motion Capture

paper_content:

Identifying a precise anatomic skeleton is important in order to ensure high quality motion capture. In this paper we discuss two skeleton fitting techniques based on 3D optical marker data. First a local technique is proposed based on relative marker trajectories. Then it is compared to a global optimization of a skeleton model. Various proposals are made to handle the skin deformation problem. Index Terms--skeleton fitting, motion capture, optical markers.
---
paper_title: An efficient method for contour tracking using active shape models

paper_content:

There has been considerable research interest recently, in the areas of real time contour tracking and active shape models. This paper demonstrates how dynamic filtering can be used in combination with a modal-based flexible shape model to track an articulated non-rigid body in motion. The results show the method being used to track the silhouette of a walking pedestrian in real time. The active shape model used was generated automatically from real image data and incorporates variability in shape due to orientation as well as object flexibility. A Kalman filter is used to control spatial scale for feature search over successive frames. Iterative refinement allows accurate contour localisation where feasible. The shape model incorporates knowledge of the likely shape of the contour and speeds up tracking by reducing the number of system parameters. A further increase in speed is obtained by filtering the shape parameters independently. >
---
paper_title: Ghost: a human body part labeling system using silhouettes

paper_content:

Ghost is a real time system for estimating human body posture and detecting body parts in monochromatic imagery. It constructs a silhouette based body model to determine the location of the body parts while people are in generic postures. It combines hierarchical body pose estimation, a convex hull analysis of the silhouette, and a partial mapping from the body parts to the silhouette segments using a distance transform method that does not violate the topology of the human body. Experimental results demonstrate robustness and real-time performance of the proposed algorithm.
---
paper_title: Human posture recognition using multi-scale morphological method and Kalman motion estimation

paper_content:

For the purpose of estimating the posture parameters of a moving human body in visual surveillance applications, we present a model-based shape analysis method and convert the problem to the optimal matching of 2D silhouettes of the human body in parametric shape space. We point out the causality of shape similarity in morphological scale space. Based on this causality and after using the /spl alpha/-/spl beta/-/spl gamma/ filter to estimate the initial value of the matching, the method we proposed works in a coarse-to-fine manner on silhouettes practically segmented from a complex environment, and can converge fast enough to meet real-time requirements.
---
paper_title: Motion-based recognition of pedestrians

paper_content:

In this paper we present an algorithm for recognizing walking pedestrians in sequences of color images taken from a moving camera. The recognition is based on the characteristic motion of the legs of a pedestrian walking parallel to the image plane. Each image is segmented into region-like image parts by clustering pixels in a combined color/position feature space. The proposed clustering technique implies matching of corresponding clusters in consecutive frames and therefore allows clusters to be tracked over a sequence of images. Based on the observation of clusters over time a two-stage classifier extracts those clusters which most likely represent the legs of pedestrians. A fast polynomial classifier performs a rough preselection of clusters by evaluating temporal changes of a shape-dependent clusters feature. The final classification is done by a time delay neural network (TDNN) with spatio-temporal receptive fields.
---
paper_title: Learning and recognizing human dynamics in video sequences

paper_content:

This paper describes a probabilistic decomposition of human dynamics at multiple abstractions, and shows how to propagate hypotheses across space, time, and abstraction levels. Recognition in this framework is the succession of very general low level grouping mechanisms to increased specific and learned model based grouping techniques at higher levels. Hard decision thresholds are delayed and resolved by higher level statistical models and temporal context. Low-level primitives are areas of coherent motion found by EM clustering, mid-level categories are simple movements represented by dynamical systems, and high-level complex gestures are represented by Hidden Markov Models as successive phases of ample movements. We show how such a representation can be learned from training data, and apply It to the example of human gait recognition.
---
paper_title: -D Reconstruction of Multipart Self-Occluding Objects

paper_content:

In this paper we present a method for reconstruction of multipart objects from several arbitrary views using deformable superquadrics as the models of the object's parts. Two visual cues are used: occluding contours and stereo (possibly aided by projected patterns). The object can be relatively complex and can exhibit numerous self occlusions from some or all views. Our preliminary experiments on a human body and a tailor's mannequin show that the reconstruction is more complete than in purely stereo or structured light based methods and more precise than the reconstruction from occluding contours only.
---
paper_title: Human Movement Analysis Based On Explicit Motion Models

paper_content:

Within the field of computer vision the automatic interpretation of human movements is one of the most challenging tasks. A central problem in analyzing such movements is due to the fact that the human body consists of body parts linked to each other at joints which allows different movements of the parts. Therefore, the human body generally has to be treated as a nonrigid or more precisely as an articulated body. In addition, for general camera positions always some of the body parts are occluded. Although occlusions can provide important cues in a recognition task, the automatic interpretation is more difficult. Another problem that has to be dealt with is the clothing which can have a large influence on the appearence of a person (wide or tight trousers, different jackets, etc.). Clothing can also cause complex illumination phenomena that, in addition, change during movement (compare with efforts in the field of computer graphics to simulate cloth objects, e.g., [83]).
---
paper_title: 3-D model-based tracking of humans in action: a multi-view approach

paper_content:

We present a vision system for the 3-D model-based tracking of unconstrained human movement. Using image sequences acquired simultaneously from multiple views, we recover the 3-D body pose at each time instant without the use of markers. The pose-recovery problem is formulated as a search problem and entails finding the pose parameters of a graphical human model whose synthesized appearance is most similar to the actual appearance of the real human in the multi-view images. The models used for this purpose are acquired from the images. We use a decomposition approach and a best-first technique to search through the high dimensional pose parameter space. A robust variant of chamfer matching is used as a fast similarity measure between synthesized and real edge images. We present initial tracking results from a large new Humans-in-Action (HIA) database containing more than 2500 frames in each of four orthogonal views. They contain subjects involved in a variety of activities, of various degrees of complexity, ranging from the more simple one-person hand waving to the challenging two-person close interaction in the Argentine Tango.
---
paper_title: Tracking hybrid 2D-3D human models from multiple views

paper_content:

A novel framework is proposed under which robust matching and tracking of a 3D skeleton model of a human body from multiple views can be performed We propose a method for measuring the ambiguity of 2D measurements provided by each view. The ambiguity measurement is then used for selecting the best view for the most accurate match and tracking. A hybrid 2D-3D representation is chosen for modelling human body poses. The hybrid model is learnt using hierarchical principal component analysis. The CONDENSATION algorithm is used to robustly track and match 3D skeleton models in individual views.
---
paper_title: Human Movement Analysis Based On Explicit Motion Models

paper_content:

Within the field of computer vision the automatic interpretation of human movements is one of the most challenging tasks. A central problem in analyzing such movements is due to the fact that the human body consists of body parts linked to each other at joints which allows different movements of the parts. Therefore, the human body generally has to be treated as a nonrigid or more precisely as an articulated body. In addition, for general camera positions always some of the body parts are occluded. Although occlusions can provide important cues in a recognition task, the automatic interpretation is more difficult. Another problem that has to be dealt with is the clothing which can have a large influence on the appearence of a person (wide or tight trousers, different jackets, etc.). Clothing can also cause complex illumination phenomena that, in addition, change during movement (compare with efforts in the field of computer graphics to simulate cloth objects, e.g., [83]).
---
paper_title: CONDENSATION—Conditional Density Propagation for Visual Tracking

paper_content:

The problem of tracking curves in dense visual clutter is challenging. Kalman filtering is inadequate because it is based on Gaussian densities which, being unimo dal, cannot represent simultaneous alternative hypotheses. The Condensation algorithm uses “factored sampling”, previously applied to the interpretation of static images, in which the probability distribution of possible interpretations is represented by a randomly generated set. Condensation uses learned dynamical models, together with visual observations, to propagate the random set over time. The result is highly robust tracking of agile motion. Notwithstanding the use of stochastic methods, the algorithm runs in near real-time.
---
paper_title: Vision-based animation of digital humans

paper_content:

The paper presents a system for animating customized virtual humans using motion parameters estimated from multi-view image sequences. The advantage of the method is that the subject does not have to wear markers or other devices. The authors offer a detailed analysis of the singularities that might arise during tracking humans using multiple cameras and provide an analysis of the criteria that allow the ordering of the cameras that provide the most information for tracking. They demonstrate their system using image sequences requiring the recovery of complex three-dimensional motions in the presence of significant occlusion, and present a quantitative analysis of the accuracy of the estimated motion parameters.
---
paper_title: A dynamic Bayesian network approach to figure tracking using learned dynamic models

paper_content:

The human figure exhibits complex and rich dynamic behavior that is both nonlinear and time-varying. However most work on tracking and synthesizing figure motion has employed either simple, generic dynamic models or highly specific hand-tailored ones. Recently, a broad class of learning and inference algorithms for time-series models have been successfully cast in the framework of dynamic Bayesian networks (DBNs). This paper describes a novel DBN-based switching linear dynamic system (SLDS) model and presents its application to figure motion analysis. A key feature of our approach is an approximate Viterbi inference technique for overcoming the intractability of exact inference in mixed-state DBNs. We present experimental results for learning figure dynamics from video data and show promising initial results for tracking, interpolation, synthesis, and classification using learned models.
---
paper_title: Multiple-view-based tracking of multiple humans

paper_content:

We propose a multiple-view-based tracking algorithm for multiple-human motions. In vision-based human tracking, self-occlusions and human-human occlusions are a part of the more significant problems. Employing multiple viewpoints and a viewpoint selection mechanism, however can reduce these problems. In our system, human positions are tracked with a sequence of multiple-viewpoint images. This tracking is based on the Kalman filtering approach. The estimation results are utilized to select proper viewpoints in other sub-tasks (rotation angle detection and body-side detection). Each sub-task has a different criterion for selecting viewpoints. We also describe the criterions for accomplishing individual sub-tasks and relationships between sub-tasks. We have already built an experimental system based on a small number of reliable image features. We confirm the stability of our algorithm through simulations. We also performed fundamental examinations on the experimental system.
---
paper_title: Multiple cues used in model-based human motion capture

paper_content:

Human motion capture has lately been the object of much attention due to commercial interests. A "touch-free" computer vision solution to the problem is desirable to avoid the intrusiveness of standard capture devices. The object to be monitored is known a priori which suggests the inclusion of a human model in the capture process. We use a model-based approach known as the analysis-by-synthesis approach. This approach is powerful but has a problem with its potential huge search space. Using multiple cues we reduce the search space by introducing constraints through the 3D locations of salient points and a silhouette of the subject. Both data types are relatively easy to derive and only require limited computational effort so the approach remains suitable for real-time applications. The approach is tested on 3D movements of a human arm and the results show that we successfully can estimate the pose of the arm using the reduced search space.
---
paper_title: A framework for modeling the appearance of 3D articulated figures

paper_content:

This paper describes a framework for constructing a linear subspace model of image appearance for complex articulated 3D figures such as humans and other animals. A commercial motion capture system provides 3D data that is aligned with images of subjects performing various activities. Portions of a limb's image appearance are seen from multiple views and for multiple subjects. From these partial views, weighted principal component analysis is used to construct a linear subspace representation of the "unwrapped" image appearance of each limb. The linear subspaces provide a generative model of the object appearance that is exploited in a Bayesian particle filtering tracking system. Results of tracking single limbs and walking humans are presented.
---
paper_title: A multiple hypothesis approach to figure tracking

paper_content:

This paper describes a probabilistic multiple-hypothesis framework for tracking highly articulated objects. In this framework, the probability density of the tracker state is represented as a set of modes with piecewise Gaussians characterizing the neighborhood around these modes. The temporal evolution of the probability density is achieved through sampling from the prior distribution, followed by local optimization of the sample positions to obtain updated modes. This method of generating hypotheses from state-space search does not require the use of discrete features unlike classical multiple-hypothesis tracking. The parametric form of the model is suited for high dimensional state-spaces which cannot be efficiently modeled using non-parametric approaches. Results are shown for tracking Fred Astaire in a movie dance sequence.
---
paper_title: The Visual Analysis of Human Movement: A Survey

paper_content:

The ability to recognize humans and their activities by vision is key for a machine to interact intelligently and effortlessly with a human-inhabited environment. Because of many potentially important applications, “looking at people” is currently one of the most active application domains in computer vision. This survey identifies a number of promising applications and provides an overview of recent developments in this domain. The scope of this survey is limited to work on whole-body or hand motion; it does not include work on human faces. The emphasis is on discussing the various methodologies; they are grouped in 2-D approaches with or without explicit shape models and 3-D approaches. Where appropriate, systems are reviewed. We conclude with some thoughts about future directions.
---
paper_title: Human motion analysis: A review

paper_content:

Human motion analysis is receiving increasing attention from computer vision researchers. This interest is motivated by a wide spectrum of applications, such as athletic performance analysis, surveillance, man?machine interfaces, content-based image storage and retrieval, and video conferencing. This paper gives an overview of the various tasks involved in motion analysis of the human body. We focus on three major areas related to interpreting human motion: (1) motion analysis involving human body parts, (2) tracking a moving human from a single view or multiple camera perspectives, and (3) recognizing human activities from image sequences. Motion analysis of human body parts involves the low-level segmentation of the human body into segments connected by joints and recovers the 3D structure of the human body using its 2D projections over a sequence of images. Tracking human motion from a single view or multiple perspectives focuses on higher-level processing, in which moving humans are tracked without identifying their body parts. After successfully matching the moving human image from one frame to another in an image sequence, understanding the human movements or activities comes naturally, which leads to our discussion of recognizing human activities.
---
paper_title: Pfinder: real-time tracking of the human body

paper_content:

Pfinder is a real-time system for tracking and interpretation of people. It runs on a standard SGI Indy computer, and has performed reliably on thousands of people in many different physical locations. The system uses a multiclass statistical model of color and shape to segment a person from a background scene, and implements heuristics which can find and track people's head and hands in a wide range of viewing conditions. Pfinder produces a real- time representation of a user useful for applications such as wireless interfaces, video databases, and low-bandwidth coding, without cumbersome wires or attached sensors.© (1996) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.
---
paper_title: Lower limb kinematics of human walking with the medial axis transformation

paper_content:

The paper describes a simple model for free-speed human walking and compares ordinary images of a walking person to the model. Three dimensional kinematic data were obtained from subjects walking with markers over the joints of their limbs. The average of these data was used to derive a model stick figure of the lower limbs, based on the average anthropometric data of the population. Stick figures were obtained from ordinary images of persons dressed in tight fitting clothes without any markers by using the medial axis transformation. The two dimensional information from the image stick figures was compared with the projection of the three dimensional information of the model onto the relevant plane. A high degree of correlation was noted between the rotational patterns of the model and image stick figures. >
---
paper_title: Multiple cues used in model-based human motion capture

paper_content:

Human motion capture has lately been the object of much attention due to commercial interests. A "touch-free" computer vision solution to the problem is desirable to avoid the intrusiveness of standard capture devices. The object to be monitored is known a priori which suggests the inclusion of a human model in the capture process. We use a model-based approach known as the analysis-by-synthesis approach. This approach is powerful but has a problem with its potential huge search space. Using multiple cues we reduce the search space by introducing constraints through the 3D locations of salient points and a silhouette of the subject. Both data types are relatively easy to derive and only require limited computational effort so the approach remains suitable for real-time applications. The approach is tested on 3D movements of a human arm and the results show that we successfully can estimate the pose of the arm using the reduced search space.
---
paper_title: Learning and synthesizing human body motion and posture

paper_content:

A novel approach is presented for estimating human body posture and motion from a video sequence. Human pose is defined as the instantaneous image plane configuration of a single articulated body in terms of the position of a predetermined set of joints. First, statistical segmentation of the human bodies from the background is performed and low-level visual features are found given the segmented body shape. The goal is to be able to map these visual features to body configurations. Given a set of body motion sequences for training, a set of clusters is built in which each has statistically similar configurations. This unsupervised task is done using the expectation maximization algorithm. Then, for each of the clusters, a neural network is trained to build this mapping. Clustering body configurations improves the mapping accuracy. Given new visual features, a mapping from each cluster is performed providing a set of possible poses. From this set, the most likely pose is extracted given the learned probability distribution and the visual feature similarity between hypothesis and input. Performance of the system is characterized using a new set of known body postures, showing promising results.
---
paper_title: Real-time estimation of human body posture from monocular thermal images

paper_content:

This paper introduces a new real-time method to estimate the posture of a human from thermal images acquired by an infrared camera regardless of the back-ground and lighting conditions. Distance transformation is performed for the human body area extracted from the thresholded thermal image for the. Calculation of the center of gravity. After the orientation of the upper half of the body is obtained by calculating the moment of inertia, significant points such as the top of the head, the tips of the hands and foot are heuristically located. In addition, the elbow and foot positions are estimated from the detected (significant) points using a genetic algorithm based learning procedure. The experimental results demonstrate the robustness of the proposed algorithm and real-time (faster than 20 frames per second) performance.
---
paper_title: Finding people by sampling

paper_content:

We show how to use a sampling method to find sparsely clad people in static images. People are modeled as an assembly of nine cylindrical segments. Segments are found using an EM algorithm and then assembled into hypotheses incrementally, using a learned likelihood model. Each assembly step passes on a set of samples of its likelihood to the next; this yields effective pruning of the space of hypotheses. The collection of available nine-segment hypotheses is then represented by a set of equivalence classes, which yield an efficient pruning process. The posterior for the number of people is obtained from the class representatives. People are counted quite accurately in images of real scenes using a MAP estimate. We show the method allows top-down as well as bottom up reasoning. While the method can be overwhelmed by very large numbers of segments, we show that this problem can be avoided by quite simple pruning steps.
---
paper_title: Model-based recognition of human posture using single synthetic images

paper_content:

Model-based vision has been predominantly concerned with the recognition of single component, rigid objects. This paper describes work attempting to recover the 3D structure of a multi-component, highly articulated object the human body. A major goal of this research has been to go beyond the level of basic object recognition, and attempt to reach a semantic level of description regarding the object's behaviour. Previous research on the recognition of human figures has assumed that the behaviour of the figure in the image is known a priori, for example, "walking", or has made use of motion information derived from image sequences. This research shows that accurate 3D structure can be recovered without such knowledge, and that descriptions of a human figure's behaviour can be obtained in terms of static posture descriptions such as; "sitting", "kneeling", "standing".
---
paper_title: Tracking human motion in an indoor environment

paper_content:

This paper presents an approach to tracking human motion in a sequence of monocular images. The process consists of detecting motion, segmenting moving subjects by recovering the background and, finally, tracking the subject of interest. The usual assumptions of small image motion, a fixed viewing system and constant velocity are systematically relaxed. Two cases are studied: (1) a viewing system with negligible motion, and (2) a viewing system with non-negligible motion.
---
paper_title: Ghost: a human body part labeling system using silhouettes

paper_content:

Ghost is a real time system for estimating human body posture and detecting body parts in monochromatic imagery. It constructs a silhouette based body model to determine the location of the body parts while people are in generic postures. It combines hierarchical body pose estimation, a convex hull analysis of the silhouette, and a partial mapping from the body parts to the silhouette segments using a distance transform method that does not violate the topology of the human body. Experimental results demonstrate robustness and real-time performance of the proposed algorithm.
---
paper_title: Model-based image analysis of human motion using constraint propagation

paper_content:

A system capable of analyzing image sequences of human motion is described. The system is structured as a ·feedback loop between high and low levels: predictions are made at the semantic level, and verifications are sought at the image level. The domain of human motion lends itself to a model-driven analysis, and the system includes a detailed model of the human body. All information extracted from the image is interpreted through a constraint network based on the structure of the human model. A constraint propagation operator is defined and its theoretical,properties developed. An implementation of this operator is described, and results of the analysis system for a short image sequence are presented.
---
paper_title: Image sequence analysis of real world human motion

paper_content:

Abstract Three fundamental problems of image sequence analysis are investigated for real world human motions. The first one is the modelling of body motion, which is approached by the methods of key frame sequence of stick figures and generalized cone approximation of body parts. The second problem is the correspondence of feature points in the consecutive frames, for which a method called “window code matching” is proposed. The last one is the analysis of occluding regions, which is studied on the bases of occlusion prediction by model and difference images.
---
paper_title: Dynamic models of human motion

paper_content:

This paper describes experiments in human motion understanding, defined here as estimation of the physical state of the body (the Plant) combined with interpretation of that part of the motion that cannot be predicted by the plant alone (the Behavior). The described behavior system operates in conjunction with a real-time, fully-dynamic, 3-D person tracking system that provides a mathematically concise formulation for incorporating a wide variety of physical constraints and probabilistic influences. The framework takes the form of a non-linear recursive filter that enables pixel-level, probabilistic processes to take advantage of the contextual knowledge encoded in the higher-level models. Results are shown that demonstrate both qualitative and quantitative gains in tracking performance.
---
paper_title: Estimation of articulated motion using kinematically constrained mixture densities

paper_content:

We address the problem of articulated posture estimation in its general form. Namely, the recovery of full 3D articulated posture parameters from an uncontrolled scene. Stochastic modeling of low-level segmented image data is unified with models of object kinematic structure through a constrained mixture of observation processes. A modified expectation-maximization algorithm is proposed for this purpose. Early experiments qualitatively demonstrate the efficacy of our approach, and provide a context for integration for more sophisticated image cues.
---
paper_title: First Sight: A human body outline labeling system

paper_content:

First Sight, a vision system in labeling the outline of a moving human body, is proposed in this paper. The emphasis of First Sight is on the analysis of motion information gathered solely from the outline of a moving human object. Two main processes are implemented in First Sight. The first process uses a novel technique to extract the outline of a moving human body from an image sequence. The second process, which employs a new human body model, interprets the outline and produces a labeled two-dimensional human body stick figure for each frame of the image sequence. Extensive knowledge of the structure, shape, and posture of the human body is used in the model. The experimental results of applying the technique on unedited image sequences with self-occlusions and missing boundary lines are encouraging. >
---
paper_title: Posture estimation using structure and motion models

paper_content:

Sensing of human motion is very important for human-computer interactive applications such as virtual reality, gesture recognition, and communication. A vision system is suitable for human-computer interaction since this involves passive sensing and the system can estimate the motion of the user without any discomfort for the user. In this paper, we propose an algorithm for fast posture estimation of the user from an image sequence by using the position information of the hands and head. Our algorithm is founded on a model based method. The parameters of a geometric model are calculated from human kinematics. It is inadequate to obtain a unique solution from the image information such as position of the head and hands. The unknown parameters are predicted by the motion models and the previous posture parameters, then the unknown parameters are adjusted by a minimization method.
---
paper_title: Tracking hybrid 2D-3D human models from multiple views

paper_content:

A novel framework is proposed under which robust matching and tracking of a 3D skeleton model of a human body from multiple views can be performed We propose a method for measuring the ambiguity of 2D measurements provided by each view. The ambiguity measurement is then used for selecting the best view for the most accurate match and tracking. A hybrid 2D-3D representation is chosen for modelling human body poses. The hybrid model is learnt using hierarchical principal component analysis. The CONDENSATION algorithm is used to robustly track and match 3D skeleton models in individual views.
---
paper_title: Human Movement Analysis Based On Explicit Motion Models

paper_content:

Within the field of computer vision the automatic interpretation of human movements is one of the most challenging tasks. A central problem in analyzing such movements is due to the fact that the human body consists of body parts linked to each other at joints which allows different movements of the parts. Therefore, the human body generally has to be treated as a nonrigid or more precisely as an articulated body. In addition, for general camera positions always some of the body parts are occluded. Although occlusions can provide important cues in a recognition task, the automatic interpretation is more difficult. Another problem that has to be dealt with is the clothing which can have a large influence on the appearence of a person (wide or tight trousers, different jackets, etc.). Clothing can also cause complex illumination phenomena that, in addition, change during movement (compare with efforts in the field of computer graphics to simulate cloth objects, e.g., [83]).
---
paper_title: Monocular tracking of the human arm in 3D

paper_content:

We address the problem of estimating the position and motion of a human arm in 3D without any constraints on its behavior and without the use of special markers. We model the arm as two truncated right-circular cones connected with spherical joints. We propose to use a recursive estimator for arm position, and to provide the estimator with error signals obtained by comparing the projected estimated arm position with that of the actual arm in the image. The system is demonstrated and tested on a real image sequence. >
---
paper_title: A dynamic Bayesian network approach to figure tracking using learned dynamic models

paper_content:

The human figure exhibits complex and rich dynamic behavior that is both nonlinear and time-varying. However most work on tracking and synthesizing figure motion has employed either simple, generic dynamic models or highly specific hand-tailored ones. Recently, a broad class of learning and inference algorithms for time-series models have been successfully cast in the framework of dynamic Bayesian networks (DBNs). This paper describes a novel DBN-based switching linear dynamic system (SLDS) model and presents its application to figure motion analysis. A key feature of our approach is an approximate Viterbi inference technique for overcoming the intractability of exact inference in mixed-state DBNs. We present experimental results for learning figure dynamics from video data and show promising initial results for tracking, interpolation, synthesis, and classification using learned models.
---
paper_title: A multiple hypothesis approach to figure tracking

paper_content:

This paper describes a probabilistic multiple-hypothesis framework for tracking highly articulated objects. In this framework, the probability density of the tracker state is represented as a set of modes with piecewise Gaussians characterizing the neighborhood around these modes. The temporal evolution of the probability density is achieved through sampling from the prior distribution, followed by local optimization of the sample positions to obtain updated modes. This method of generating hypotheses from state-space search does not require the use of discrete features unlike classical multiple-hypothesis tracking. The parametric form of the model is suited for high dimensional state-spaces which cannot be efficiently modeled using non-parametric approaches. Results are shown for tracking Fred Astaire in a movie dance sequence.
---
paper_title: Pfinder: real-time tracking of the human body

paper_content:

Pfinder is a real-time system for tracking and interpretation of people. It runs on a standard SGI Indy computer, and has performed reliably on thousands of people in many different physical locations. The system uses a multiclass statistical model of color and shape to segment a person from a background scene, and implements heuristics which can find and track people's head and hands in a wide range of viewing conditions. Pfinder produces a real- time representation of a user useful for applications such as wireless interfaces, video databases, and low-bandwidth coding, without cumbersome wires or attached sensors.© (1996) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.
---
paper_title: Cardboard people: a parameterized model of articulated image motion

paper_content:

We extend the work of Black and Yacoob (1995) on the tracking and recognition of human facial expressions using parametrized models of optical flow to deal with the articulated motion of human limbs. We define a "card-board person model" in which a person's limbs are represented by a set of connected planar patches. The parametrized image motion of these patches in constrained to enforce articulated motion and is solved for directly using a robust estimation technique. The recovered motion parameters provide a rich and concise description of the activity that can be used for recognition. We propose a method for performing view-based recognition of human activities from the optical flow parameters that extends previous methods to cope with the cyclical nature of human motion. We illustrate the method with examples of tracking human legs of long image sequences.
---
paper_title: Human Movement Analysis Based On Explicit Motion Models

paper_content:

Within the field of computer vision the automatic interpretation of human movements is one of the most challenging tasks. A central problem in analyzing such movements is due to the fact that the human body consists of body parts linked to each other at joints which allows different movements of the parts. Therefore, the human body generally has to be treated as a nonrigid or more precisely as an articulated body. In addition, for general camera positions always some of the body parts are occluded. Although occlusions can provide important cues in a recognition task, the automatic interpretation is more difficult. Another problem that has to be dealt with is the clothing which can have a large influence on the appearence of a person (wide or tight trousers, different jackets, etc.). Clothing can also cause complex illumination phenomena that, in addition, change during movement (compare with efforts in the field of computer graphics to simulate cloth objects, e.g., [83]).
---
paper_title: A model driven 3D image interpretation system applied to person detection in video images

paper_content:

Detecting persons in images is up to now an unsolved problem in general. We propose an approach which strictly follows the assumption that only objects and parts of these objects can be detected, if their appearance is known in advance. We consider the detection step as a first step towards the recognition process. The latter contains the reasoning about the movements of a person. The appearance of a 3D object means the description of the mapping of object features into the image. The proposed system uses a hierarchical model to control the image segmentation and matching process. It is a multipurpose model for articulated objects which is supporting the linkage of different features.
---
paper_title: Model-based recognition of human posture using single synthetic images

paper_content:

Model-based vision has been predominantly concerned with the recognition of single component, rigid objects. This paper describes work attempting to recover the 3D structure of a multi-component, highly articulated object the human body. A major goal of this research has been to go beyond the level of basic object recognition, and attempt to reach a semantic level of description regarding the object's behaviour. Previous research on the recognition of human figures has assumed that the behaviour of the figure in the image is known a priori, for example, "walking", or has made use of motion information derived from image sequences. This research shows that accurate 3D structure can be recovered without such knowledge, and that descriptions of a human figure's behaviour can be obtained in terms of static posture descriptions such as; "sitting", "kneeling", "standing".
---
paper_title: Extraction of parametric human model for posture recognition using genetic algorithm

paper_content:

We present in this paper an approach to extracting a human parametric 2D model for the purpose of estimating human posture and recognizing human activity. This task is done in two steps. In the first step, a human silhouette is extracted from a complex background under a fixed camera through a statistical method. By this method, we can reconstruct the background dynamically and obtain the moving silhouette. In the second step, a genetic algorithm is used to match the silhouette of the human body to a model in parametric shape space. In order to reduce the searching dimension, a layer method is proposed to take the advantage of the human model. Additionally we apply a structure-oriented Kalman filter to estimate the motion of body parts. Therefore the initial population and value in the GA can be well constrained. Experiments on real video sequences show that our method can extract the human model robustly and accurately.
---
paper_title: Vision-based animation of digital humans

paper_content:

The paper presents a system for animating customized virtual humans using motion parameters estimated from multi-view image sequences. The advantage of the method is that the subject does not have to wear markers or other devices. The authors offer a detailed analysis of the singularities that might arise during tracking humans using multiple cameras and provide an analysis of the criteria that allow the ordering of the cameras that provide the most information for tracking. They demonstrate their system using image sequences requiring the recovery of complex three-dimensional motions in the presence of significant occlusion, and present a quantitative analysis of the accuracy of the estimated motion parameters.
---
paper_title: Automated body modeling from video sequences

paper_content:

Synthetic modeling of human bodies and the simulation of motion is a long-standing problem in animation and much work is involved before a near-realistic performance can be achieved. At present, it takes an experienced designer a very long time to build a complete and realistic model that closely resembles a specific person. Our ultimate goal is to automate the process and to produce realistic animation models given a set of video sequences. In this paper we show that, given video sequences of a person moving in front of the camera, we can recover shape information and joint locations. Both of which are essential to instantiate a complete and realistic model that closely resembles a specific person and without knowledge about the position of the articulations a character cannot be animated. This is achieved with minimal human intervention. The recovered shape and motion parameters can be used to reconstruct the original movement or to allow other animation models to mimic the subject's actions.
---
paper_title: 3-D model-based tracking of humans in action: a multi-view approach

paper_content:

We present a vision system for the 3-D model-based tracking of unconstrained human movement. Using image sequences acquired simultaneously from multiple views, we recover the 3-D body pose at each time instant without the use of markers. The pose-recovery problem is formulated as a search problem and entails finding the pose parameters of a graphical human model whose synthesized appearance is most similar to the actual appearance of the real human in the multi-view images. The models used for this purpose are acquired from the images. We use a decomposition approach and a best-first technique to search through the high dimensional pose parameter space. A robust variant of chamfer matching is used as a fast similarity measure between synthesized and real edge images. We present initial tracking results from a large new Humans-in-Action (HIA) database containing more than 2500 frames in each of four orthogonal views. They contain subjects involved in a variety of activities, of various degrees of complexity, ranging from the more simple one-person hand waving to the challenging two-person close interaction in the Argentine Tango.
---
paper_title: A framework for modeling the appearance of 3D articulated figures

paper_content:

This paper describes a framework for constructing a linear subspace model of image appearance for complex articulated 3D figures such as humans and other animals. A commercial motion capture system provides 3D data that is aligned with images of subjects performing various activities. Portions of a limb's image appearance are seen from multiple views and for multiple subjects. From these partial views, weighted principal component analysis is used to construct a linear subspace representation of the "unwrapped" image appearance of each limb. The linear subspaces provide a generative model of the object appearance that is exploited in a Bayesian particle filtering tracking system. Results of tracking single limbs and walking humans are presented.
---
paper_title: Knowledge-guided visual perception of 3-D human gait from a single image sequence

paper_content:

A computer vision method is presented to determine the 3-D spatial locations of joints or feature points of human body from a film recording the human motion during walking. The proposed method first applies the geometric projection theory to obtain a set of feasible postures from a single image, then it makes use of the given dimensions of the human stick figure, physiological and motion-specific knowledge to constrain the feasible postures in both the single-frame analysis and the multi-frame analysis. Finally a unique gait interpretation is selected by an optimization algorithm. Computer simulations are used to illustrate the ideas presented. >
---
paper_title: Human motion analysis based on a robot arm model

paper_content:

A model-based method for analyzing a human body motion is presented. The method is based on a robot arm model which represents a human body motion. Combining the model and the gradient scheme, the movement of the configuration of the model (Human) can be directly estimated from an image sequence. >
---
paper_title: Tracking people with twists and exponential maps

paper_content:

This paper demonstrates a new visual motion estimation technique that is able to recover high degree-of-freedom articulated human body configurations in complex video sequences. We introduce the use of a novel mathematical technique, the product of exponential maps and twist motions, and its integration into a differential motion estimation. This results in solving simple linear systems, and enables us to recover robustly the kinematic degrees-of-freedom in noise and complex self occluded configurations. We demonstrate this on several image sequences of people doing articulated full body movements, and visualize the results in re-animating an artificial 3D human model. We are also able to recover and re-animate the famous movements of Eadweard Muybridge's motion studies from the last century. To the best of our knowledge, this is the first computer vision based system that is able to process such challenging footage and recover complex motions with such high accuracy.
---
paper_title: Dynamic models of human motion

paper_content:

This paper describes experiments in human motion understanding, defined here as estimation of the physical state of the body (the Plant) combined with interpretation of that part of the motion that cannot be predicted by the plant alone (the Behavior). The described behavior system operates in conjunction with a real-time, fully-dynamic, 3-D person tracking system that provides a mathematically concise formulation for incorporating a wide variety of physical constraints and probabilistic influences. The framework takes the form of a non-linear recursive filter that enables pixel-level, probabilistic processes to take advantage of the contextual knowledge encoded in the higher-level models. Results are shown that demonstrate both qualitative and quantitative gains in tracking performance.
---
paper_title: An automatic rotoscopy system for human motion based on a biomechanic graphical model

paper_content:

Abstract A system for analysis and synthesis of human motion is presented. The whole system consists of an analysis part and a synthesis part. The analysis part includes the preprocessing phase, modeling phase, matching phase, and interpretation phase. The synthesis part reconstructs the same motion of the human body. A biomechanic graphical model is defined to represent the human body in 3D space which is matched with the real images to discover the motion. The synthesis part uses the result of the analysis part to reconstruct the same motion of the human body which can be viewed from any viewpoint and from any distance.
---
paper_title: Human body limbs tracking by multi-ocular vision

paper_content:

This article describes a tracking method of human body limbs from multi-ocular sequence of perspective images. These limbs and the associated articulations must be first modelled. During a learning step, we model the texture linked to the limbs. The lack of characteristic points on the skin is compensated by the wearing a non-repetitive texture tight. The principle of the method is based on the interpretation of image textured patterns as the SD perspective projections of points of the textured articulated model. An iterative Levenberg-Marquardt process is used to compute the model pose in accordance with the analysed image. The calculated attitude is filtered to predict the model pose in the following image of the sequence. The image patterns are extracted locally according to the textured articulated model in the predicted attitude. Tracking experiments, illustrated in this paper with cycling sequences, demonstrate the validity of the approach.
---
paper_title: -D Reconstruction of Multipart Self-Occluding Objects

paper_content:

In this paper we present a method for reconstruction of multipart objects from several arbitrary views using deformable superquadrics as the models of the object's parts. Two visual cues are used: occluding contours and stereo (possibly aided by projected patterns). The object can be relatively complex and can exhibit numerous self occlusions from some or all views. Our preliminary experiments on a human body and a tailor's mannequin show that the reconstruction is more complete than in purely stereo or structured light based methods and more precise than the reconstruction from occluding contours only.
---
paper_title: Model-based vision: a program to see a walking person☆

paper_content:

Abstract For a machine to be able to ‘see’, it must know something about the object it is ‘looking’ at. A common method in machine vision is to provide the machine with general rather than specific knowledge about the object. An alternative technique, and the one used in this paper, is a model-based approach in which particulars about the object are given and this drives the analysis. The computer program described here, the WALKER model, maps images into a description in which a person is represented by the series of hierarchical levels, i.e. a person has an arm which has a lower-arm which has a hand. The performance of the program is illustrated by superimposing the machine-generated picture over the original photographic images.
---
paper_title: Vision-based animation of digital humans

paper_content:

The paper presents a system for animating customized virtual humans using motion parameters estimated from multi-view image sequences. The advantage of the method is that the subject does not have to wear markers or other devices. The authors offer a detailed analysis of the singularities that might arise during tracking humans using multiple cameras and provide an analysis of the criteria that allow the ordering of the cameras that provide the most information for tracking. They demonstrate their system using image sequences requiring the recovery of complex three-dimensional motions in the presence of significant occlusion, and present a quantitative analysis of the accuracy of the estimated motion parameters.
---
paper_title: Monocular tracking of the human arm in 3D

paper_content:

We address the problem of estimating the position and motion of a human arm in 3D without any constraints on its behavior and without the use of special markers. We model the arm as two truncated right-circular cones connected with spherical joints. We propose to use a recursive estimator for arm position, and to provide the estimator with error signals obtained by comparing the projected estimated arm position with that of the actual arm in the image. The system is demonstrated and tested on a real image sequence. >
---
paper_title: Model-based estimation of 3D human motion with occlusion based on active multi-viewpoint selection

paper_content:

We present a new method for the 3D model-based tracking of human body parts. To mitigate the difficulties arising due to occlusion among body parts, we employ multiple calibrated cameras in a mutually orthogonal configuration. In addition, we develop criteria for a time varying active selection of a set of cameras to track the motion of a particular human part. In particular, at every frame, each camera tracks a number of parts depending on the visibility of these parts and the observability of their predicted motion from the specific camera. To relate points on the occluding contours of the parts to points on their models we apply concepts from projective geometry. Then, within the physics-based framework we compute the generalized forces applied from the parts' occluding contours to model points of the body parts. These forces update the translational and rotational degrees of freedom of the model, such as to minimize the discrepancy between the sensory data and the estimated model state. We present initial tracking results from a series of experiments involving the recovery of complex 3D motions in the presence of significant occlusion.
---
paper_title: Automated body modeling from video sequences

paper_content:

Synthetic modeling of human bodies and the simulation of motion is a long-standing problem in animation and much work is involved before a near-realistic performance can be achieved. At present, it takes an experienced designer a very long time to build a complete and realistic model that closely resembles a specific person. Our ultimate goal is to automate the process and to produce realistic animation models given a set of video sequences. In this paper we show that, given video sequences of a person moving in front of the camera, we can recover shape information and joint locations. Both of which are essential to instantiate a complete and realistic model that closely resembles a specific person and without knowledge about the position of the articulations a character cannot be animated. This is achieved with minimal human intervention. The recovered shape and motion parameters can be used to reconstruct the original movement or to allow other animation models to mimic the subject's actions.
---
paper_title: 3-D model-based tracking of humans in action: a multi-view approach

paper_content:

We present a vision system for the 3-D model-based tracking of unconstrained human movement. Using image sequences acquired simultaneously from multiple views, we recover the 3-D body pose at each time instant without the use of markers. The pose-recovery problem is formulated as a search problem and entails finding the pose parameters of a graphical human model whose synthesized appearance is most similar to the actual appearance of the real human in the multi-view images. The models used for this purpose are acquired from the images. We use a decomposition approach and a best-first technique to search through the high dimensional pose parameter space. A robust variant of chamfer matching is used as a fast similarity measure between synthesized and real edge images. We present initial tracking results from a large new Humans-in-Action (HIA) database containing more than 2500 frames in each of four orthogonal views. They contain subjects involved in a variety of activities, of various degrees of complexity, ranging from the more simple one-person hand waving to the challenging two-person close interaction in the Argentine Tango.
---
paper_title: Tracking people with twists and exponential maps

paper_content:

This paper demonstrates a new visual motion estimation technique that is able to recover high degree-of-freedom articulated human body configurations in complex video sequences. We introduce the use of a novel mathematical technique, the product of exponential maps and twist motions, and its integration into a differential motion estimation. This results in solving simple linear systems, and enables us to recover robustly the kinematic degrees-of-freedom in noise and complex self occluded configurations. We demonstrate this on several image sequences of people doing articulated full body movements, and visualize the results in re-animating an artificial 3D human model. We are also able to recover and re-animate the famous movements of Eadweard Muybridge's motion studies from the last century. To the best of our knowledge, this is the first computer vision based system that is able to process such challenging footage and recover complex motions with such high accuracy.
---
paper_title: -D Reconstruction of Multipart Self-Occluding Objects

paper_content:

In this paper we present a method for reconstruction of multipart objects from several arbitrary views using deformable superquadrics as the models of the object's parts. Two visual cues are used: occluding contours and stereo (possibly aided by projected patterns). The object can be relatively complex and can exhibit numerous self occlusions from some or all views. Our preliminary experiments on a human body and a tailor's mannequin show that the reconstruction is more complete than in purely stereo or structured light based methods and more precise than the reconstruction from occluding contours only.
---
paper_title: Human motion analysis: A review

paper_content:

Human motion analysis is receiving increasing attention from computer vision researchers. This interest is motivated by a wide spectrum of applications, such as athletic performance analysis, surveillance, man?machine interfaces, content-based image storage and retrieval, and video conferencing. This paper gives an overview of the various tasks involved in motion analysis of the human body. We focus on three major areas related to interpreting human motion: (1) motion analysis involving human body parts, (2) tracking a moving human from a single view or multiple camera perspectives, and (3) recognizing human activities from image sequences. Motion analysis of human body parts involves the low-level segmentation of the human body into segments connected by joints and recovers the 3D structure of the human body using its 2D projections over a sequence of images. Tracking human motion from a single view or multiple perspectives focuses on higher-level processing, in which moving humans are tracked without identifying their body parts. After successfully matching the moving human image from one frame to another in an image sequence, understanding the human movements or activities comes naturally, which leads to our discussion of recognizing human activities.
---
paper_title: Upside-down presentation of the Johansson moving light-spot pattern.

paper_content:

In a film produced by Johansson, a group of moving spots, corresponding to lights attached to the main joints of a walker or a runner, gives instantly a vivid impression of a person walking or running. Even when this film was inverted and run backward, some sort of human movement was still perceived. It was perceived more frequently as an upright image of a person moving forward in a very strange manner than as an inverted image of a person moving backward. Such strangeness seemed to arise from the fact that the actor's arms were perceived as legs and vice versa. The phase relations typical of the Johansson pattern are therefore still present when the film is inverted and run backward, leading to the perception of biological motions though these motions are very strange.
---
paper_title: Virtual Stage: a location-based karaoke system

paper_content:

Virtual Stage is an extended karaoke system based on virtual reality technology. A participant becomes the main character in an interactive music video. While singing along with the song played by Virtual Stage, the participant uses body movements to interact with the virtual characters, which reside in a virtual environment and behave intelligently. In Virtual Stage, the participant's video image, grabbed by a video camera, is composited with the rendered image of the 3D virtual environment. Participants will see the realistic video image of themselves and get a false feeling of immersion, as if existing in the VE.
---
paper_title: Model-based recognition of human posture using single synthetic images

paper_content:

Model-based vision has been predominantly concerned with the recognition of single component, rigid objects. This paper describes work attempting to recover the 3D structure of a multi-component, highly articulated object the human body. A major goal of this research has been to go beyond the level of basic object recognition, and attempt to reach a semantic level of description regarding the object's behaviour. Previous research on the recognition of human figures has assumed that the behaviour of the figure in the image is known a priori, for example, "walking", or has made use of motion information derived from image sequences. This research shows that accurate 3D structure can be recovered without such knowledge, and that descriptions of a human figure's behaviour can be obtained in terms of static posture descriptions such as; "sitting", "kneeling", "standing".
---
paper_title: Using phase space constraints to represent human body motion

paper_content:

A regulated power supply system is disclosed including a source of unidirectional electric current, and at least one switch device including a saturable reactor controlled between a saturated condition and an unsaturated condition in a duty cycle such that the load voltage tends to remain constant with varying load current. In one embodiment, a pair of switch devices are coupled, respectively, between the ends of a center-tapped secondary winding of a transformer having a primary winding fed from an inverter, and a pair of rectifier devices coupled to provide the unidirectional current to the load. Control of the switch devices is effected by applying an appropriately varying control voltage to like electrodes of respective inverted control diodes each coupled to the junction between a respective switch device and its associated rectifier device. The control diodes are driven from the output of a comparator amplifier responsive to the difference between the load voltage and a reference voltage. A pulse modulator fed by the amplifier output provides the control diodes with pulses of appropriate varying duty cycle.
---
paper_title: Ghost: a human body part labeling system using silhouettes

paper_content:

Ghost is a real time system for estimating human body posture and detecting body parts in monochromatic imagery. It constructs a silhouette based body model to determine the location of the body parts while people are in generic postures. It combines hierarchical body pose estimation, a convex hull analysis of the silhouette, and a partial mapping from the body parts to the silhouette segments using a distance transform method that does not violate the topology of the human body. Experimental results demonstrate robustness and real-time performance of the proposed algorithm.
---
paper_title: Stereoscopic system for human body tracking in natural scenes

paper_content:

Human body detection and tracking in a scene constitutes a very active working field clue to their applicability to many areas, specially as a man-machine interface (MMI) means. The system presented aims to improve the reliability and efficiency of teleoperation. The system is of application to teleoperated manipulation in civil applications such as big robots in shipyards, mines, public works or even cranes. Image segmentation is performed from movement detection. The recognition of moving bodies is verified by means of a simplified articulated cylindrical model, thus allowing to operate with a low computational cost.
---
paper_title: Computer vision for computer games

paper_content:

The appeal of computer games may be enhanced by vision-based user inputs. The high speed and low cost requirements for near-term, mass-market game applications make system design challenging. The response time of the vision interface should be less than a video frame time and the interface should cost less than $50 U.S. We meet these constraints with algorithms tailored to particular hardware. We have developed a special detector, called the artificial retina chip, which allows for fast, on-chip image processing. We describe two algorithms, based on image moments and orientation histograms, which exploit the capabilities of the chip to provide interactive response to the player's hand or body positions at 10 msec frame time and at low-cost. We show several possible game interactions.
---
paper_title: Gesture recognition using the Perseus architecture

paper_content:

Communication involves more than simply spoken information. Typical interactions use gestures to accurately and efficiently convey ideas that are more easily expressed with actions than words. A more intuitive interface with machines should involve not only speech recognition, but gesture recognition as well. One of the most frequently used and expressively powerful gestures is pointing. It is far easier and more accurate to point to an object than give a verbal description of its location. To produce a more efficient, accurate, and natural human-machine interface we use the Perseus architecture to interpret the pointing gesture. Perseus uses a variety of techniques to reliably solve this complex visual problem in non-engineered worlds. Knowledge about the task and environment is used at all stages of processing to best interpret the scene for the current situation. Once the visual operators are chosen, contextual knowledge is used to tune them for maximal performance. Redundant interpretation of the scene provides robustness to errors in interpretation. Fusion of independent types of information results in increased tolerance when assumptions about the environment fail. Windows of attention are used to improve speed and remove distractions from the scene. Furthermore, reuse is a major issue in the design of Perseus. Information about the environment and task is explicitly represented so it can easily be re-used in tasks other than pointing. A clean interface to Perseus is provided for symbolic higher level systems like the RAP reactive execution system. In this paper we describe Perseus in detail and show how it is used to locate objects pointed to by people.
---
paper_title: Pedestrian detection using wavelet templates

paper_content:

This paper presents a trainable object detection architecture that is applied to detecting people in static images of cluttered scenes. This problem poses several challenges. People are highly non-rigid objects with a high degree of variability in size, shape, color, and texture. Unlike previous approaches, this system learns from examples and does not rely on any a priori (hand-crafted) models or on motion. The detection technique is based on the novel idea of the wavelet template that defines the shape of an object in terms of a subset of the wavelet coefficients of the image. It is invariant to changes in color and texture and can be used to robustly define a rich and complex class of objects such as people. We show how the invariant properties and computational efficiency of the wavelet template make it an effective tool for object detection.
---
paper_title: Detection and estimation of pointing gestures in dense disparity maps

paper_content:

We describe a real-time system for detecting pointing gestures and estimating the direction of pointing using stereo cameras. Previously, similar systems were implemented using color-based blob trackers, which relied on effective skin color detection; this approach is sensitive to lighting changes and the clothing worn by the user. In contrast, we used a stereo system that produces dense disparity maps in real-time. Disparity maps are considerably less sensitive to lighting changes. Our system subtracts the background, analyzes the foreground pixels to break the body into parts using a robust mixture model, and estimates the direction of pointing. We have tested the system on both coarse and fine pointing by selecting the targets in a room and controlling the cursor on a wall screen, respectively.
---
paper_title: Classification of human body motion

paper_content:

The classification of human body motion is a difficult problem. In particular, the automatic segmentation of image sequences containing more than one class of motion is challenging. An effective approach is to use mixed discrete/continuous states to couple perception with classification. A spline contour is used to track the outline of the person. We show that, for a quasi-periodic human body motion, an autoregressive process is a suitable model for the contour dynamics. This can then be used as a dynamical model for mixed-state "condensation" filtering, switching automatically between different motion classes. We have developed "partial importance sampling" to enhance the efficiency of the mixed-state condensation filter. It is also shown that the importance sampling can be done in linear time, instead of the previous quadratic algorithm. "Tying" of discrete states is used to obtain further efficiency improvements. Automatic segmentation is demonstrated on video sequences of aerobic exercises. The performance is promising, but there remains a residual misclassification rate, and possible explanations for this are discussed.
---
paper_title: Cardboard people: a parameterized model of articulated image motion

paper_content:

We extend the work of Black and Yacoob (1995) on the tracking and recognition of human facial expressions using parametrized models of optical flow to deal with the articulated motion of human limbs. We define a "card-board person model" in which a person's limbs are represented by a set of connected planar patches. The parametrized image motion of these patches in constrained to enforce articulated motion and is solved for directly using a robust estimation technique. The recovered motion parameters provide a rich and concise description of the activity that can be used for recognition. We propose a method for performing view-based recognition of human activities from the optical flow parameters that extends previous methods to cope with the cyclical nature of human motion. We illustrate the method with examples of tracking human legs of long image sequences.
---
paper_title: Human Movement Analysis Based On Explicit Motion Models

paper_content:

Within the field of computer vision the automatic interpretation of human movements is one of the most challenging tasks. A central problem in analyzing such movements is due to the fact that the human body consists of body parts linked to each other at joints which allows different movements of the parts. Therefore, the human body generally has to be treated as a nonrigid or more precisely as an articulated body. In addition, for general camera positions always some of the body parts are occluded. Although occlusions can provide important cues in a recognition task, the automatic interpretation is more difficult. Another problem that has to be dealt with is the clothing which can have a large influence on the appearence of a person (wide or tight trousers, different jackets, etc.). Clothing can also cause complex illumination phenomena that, in addition, change during movement (compare with efforts in the field of computer graphics to simulate cloth objects, e.g., [83]).
---
paper_title: Real-time human motion analysis by image skeletonization

paper_content:

In this paper a process is described for analysing the motion of a human target in a video stream. Moving targets are detected and their boundaries extracted. From these, a "star" skeleton is produced. Two motion cues are determined from this skeletonization: body posture, and cyclic motion of skeleton segments. These cues are used to determine human activities such as walking or running, and even potentially, the target's gait. Unlike other methods, this does not require an a priori human model, or a large number of "pixels on target". Furthermore, it is computationally inexpensive, and thus ideal for real-world video applications such as outdoor video surveillance.
---
paper_title: Backpack : Detection of People Carrying Objects Using Silhouettes

paper_content:

Abstract We describe a video-rate surveillance algorithm for determining whether people are carrying objects or moving unencumbered from a stationary camera. The contribution of the paper is the shape analysis algorithm that both determines whether a person is carrying an object and segments the object from the person so that it can be tracked, e.g., during an exchange of objects between two people. As the object is segmented, an appearance model of the object is constructed. The method combines periodic motion estimation with static symmetry analysis of the silhouettes of a person in each frame of the sequence. Experimental results demonstrate robustness and real-time performance of the proposed algorithm.
---
paper_title: Analyzing and recognizing walking figures in XYT

paper_content:

We describe a novel algorithm for gait analysis. A person walking frontoparallel to the image plane generates a characteristic "braided" pattern in a spatiotemporal (XYT) volume. Our algorithm detects this pattern, and fits it with a set of spatiotemporal snakes. The snakes can be used to find the bounding contours of the walker. The contours vary over time in a manner characteristic of each walker. Individual gaits can be recognized by applying standard pattern recognition techniques to the contour signals. >
---
paper_title: A dynamic Bayesian network approach to figure tracking using learned dynamic models

paper_content:

The human figure exhibits complex and rich dynamic behavior that is both nonlinear and time-varying. However most work on tracking and synthesizing figure motion has employed either simple, generic dynamic models or highly specific hand-tailored ones. Recently, a broad class of learning and inference algorithms for time-series models have been successfully cast in the framework of dynamic Bayesian networks (DBNs). This paper describes a novel DBN-based switching linear dynamic system (SLDS) model and presents its application to figure motion analysis. A key feature of our approach is an approximate Viterbi inference technique for overcoming the intractability of exact inference in mixed-state DBNs. We present experimental results for learning figure dynamics from video data and show promising initial results for tracking, interpolation, synthesis, and classification using learned models.
---
paper_title: Recognition of human body motion using phase space constraints

paper_content:

A new method for representing and recognizing human body movements is presented. The basic idea is to identify sets of constraints that are diagnostic of a movement: expressed using body-centered coordinates such as joint angles and in force only during a particular movement. Assuming the availability of Cartesian tracking data, we develop techniques for a representation of movements defined by space curves in subspaces of a "phase space." The phase space has axes of joint angles and torso location and attitude, and the axes of the subspaces are subsets of the axes of the phase space. Using this representation we develop a system for learning new movements from ground truth data by searching for constraints. We then use the learned representation for recognizing movements in unsegmented data. We train and test the system on nine fundamental steps from classical ballet performed by two dancers; the system accurately recognizes the movements in the unsegmented stream of motion. >
---
paper_title: Lower limb kinematics of human walking with the medial axis transformation

paper_content:

The paper describes a simple model for free-speed human walking and compares ordinary images of a walking person to the model. Three dimensional kinematic data were obtained from subjects walking with markers over the joints of their limbs. The average of these data was used to derive a model stick figure of the lower limbs, based on the average anthropometric data of the population. Stick figures were obtained from ordinary images of persons dressed in tight fitting clothes without any markers by using the medial axis transformation. The two dimensional information from the image stick figures was compared with the projection of the three dimensional information of the model onto the relevant plane. A high degree of correlation was noted between the rotational patterns of the model and image stick figures. >
---
paper_title: Visual pattern recognition by moment invariants

paper_content:

In this paper a theory of two-dimensional moment invariants for planar geometric figures is presented. A fundamental theorem is established to relate such moment invariants to the well-known algebraic invariants. Complete systems of moment invariants under translation, similitude and orthogonal transformations are derived. Some moment invariants under general two-dimensional linear transformations are also included. Both theoretical formulation and practical models of visual pattern recognition based upon these moment invariants are discussed. A simple simulation program together with its performance are also presented. It is shown that recognition of geometrical patterns and alphabetical characters independently of position, size and orientation can be accomplished. It is also indicated that generalization is possible to include invariance with parallel projection.
---
paper_title: Stereoscopic system for human body tracking in natural scenes

paper_content:

Human body detection and tracking in a scene constitutes a very active working field clue to their applicability to many areas, specially as a man-machine interface (MMI) means. The system presented aims to improve the reliability and efficiency of teleoperation. The system is of application to teleoperated manipulation in civil applications such as big robots in shipyards, mines, public works or even cranes. Image segmentation is performed from movement detection. The recognition of moving bodies is verified by means of a simplified articulated cylindrical model, thus allowing to operate with a low computational cost.
---
paper_title: Motion-based recognition of pedestrians

paper_content:

In this paper we present an algorithm for recognizing walking pedestrians in sequences of color images taken from a moving camera. The recognition is based on the characteristic motion of the legs of a pedestrian walking parallel to the image plane. Each image is segmented into region-like image parts by clustering pixels in a combined color/position feature space. The proposed clustering technique implies matching of corresponding clusters in consecutive frames and therefore allows clusters to be tracked over a sequence of images. Based on the observation of clusters over time a two-stage classifier extracts those clusters which most likely represent the legs of pedestrians. A fast polynomial classifier performs a rough preselection of clusters by evaluating temporal changes of a shape-dependent clusters feature. The final classification is done by a time delay neural network (TDNN) with spatio-temporal receptive fields.
---
paper_title: Tracking interacting people

paper_content:

A computer vision system for tracking multiple people in relatively unconstrained environments is described. Tracking is performed at three levels of abstraction: regions, people and groups. A novel, adaptive background subtraction method that combines colour and gradient information is used to cope with shadows and unreliable colour cues. People are tracked through mutual occlusions as they form groups and part from one another. Strong use is made of colour information to disambiguate occlusions and to provide qualitative estimates of depth ordering and position during occlusion. Some simple interactions with objects can also be detected. The system is tested using indoor and outdoor sequences. It is robust and should provide a useful mechanism for bootstrapping and reinitialisation of tracking using more-specific but less-robust human models.
---
paper_title: Learning and recognizing human dynamics in video sequences

paper_content:

This paper describes a probabilistic decomposition of human dynamics at multiple abstractions, and shows how to propagate hypotheses across space, time, and abstraction levels. Recognition in this framework is the succession of very general low level grouping mechanisms to increased specific and learned model based grouping techniques at higher levels. Hard decision thresholds are delayed and resolved by higher level statistical models and temporal context. Low-level primitives are areas of coherent motion found by EM clustering, mid-level categories are simple movements represented by dynamical systems, and high-level complex gestures are represented by Hidden Markov Models as successive phases of ample movements. We show how such a representation can be learned from training data, and apply It to the example of human gait recognition.
---
paper_title: Parameterized Modeling and Recognition of Activities

paper_content:

In this paper we consider a class of human activities?atomic activities?which can be represented as a set of measurements over a finite temporal window (e.g., the motion of human body parts during a walking cycle) and which has a relatively small space of variations in performance. A new approach for modeling and recognition of atomic activities that employs principal component analysis and analytical global transformations is proposed. The modeling of sets of exemplar instances of activities that are similar in duration and involve similar body part motions is achieved by parameterizing their representation using principal component analysis. The recognition of variants of modeled activities is achieved by searching the space of admissible parameterized transformations that these activities can undergo. This formulation iteratively refines the recognition of the class to which the observed activity belongs and the transformation parameters that relate it to the model in its class. We provide several experiments on recognition of articulated and deformable human motions from image motion parameters.
---
paper_title: Backpack : Detection of People Carrying Objects Using Silhouettes

paper_content:

Abstract We describe a video-rate surveillance algorithm for determining whether people are carrying objects or moving unencumbered from a stationary camera. The contribution of the paper is the shape analysis algorithm that both determines whether a person is carrying an object and segments the object from the person so that it can be tracked, e.g., during an exchange of objects between two people. As the object is segmented, an appearance model of the object is constructed. The method combines periodic motion estimation with static symmetry analysis of the silhouettes of a person in each frame of the sequence. Experimental results demonstrate robustness and real-time performance of the proposed algorithm.
---
paper_title: Pfinder: real-time tracking of the human body

paper_content:

Pfinder is a real-time system for tracking and interpretation of people. It runs on a standard SGI Indy computer, and has performed reliably on thousands of people in many different physical locations. The system uses a multiclass statistical model of color and shape to segment a person from a background scene, and implements heuristics which can find and track people's head and hands in a wide range of viewing conditions. Pfinder produces a real- time representation of a user useful for applications such as wireless interfaces, video databases, and low-bandwidth coding, without cumbersome wires or attached sensors.© (1996) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.
---
paper_title: 3-D model-based tracking of humans in action: a multi-view approach

paper_content:

We present a vision system for the 3-D model-based tracking of unconstrained human movement. Using image sequences acquired simultaneously from multiple views, we recover the 3-D body pose at each time instant without the use of markers. The pose-recovery problem is formulated as a search problem and entails finding the pose parameters of a graphical human model whose synthesized appearance is most similar to the actual appearance of the real human in the multi-view images. The models used for this purpose are acquired from the images. We use a decomposition approach and a best-first technique to search through the high dimensional pose parameter space. A robust variant of chamfer matching is used as a fast similarity measure between synthesized and real edge images. We present initial tracking results from a large new Humans-in-Action (HIA) database containing more than 2500 frames in each of four orthogonal views. They contain subjects involved in a variety of activities, of various degrees of complexity, ranging from the more simple one-person hand waving to the challenging two-person close interaction in the Argentine Tango.
---
paper_title: Learning and synthesizing human body motion and posture

paper_content:

A novel approach is presented for estimating human body posture and motion from a video sequence. Human pose is defined as the instantaneous image plane configuration of a single articulated body in terms of the position of a predetermined set of joints. First, statistical segmentation of the human bodies from the background is performed and low-level visual features are found given the segmented body shape. The goal is to be able to map these visual features to body configurations. Given a set of body motion sequences for training, a set of clusters is built in which each has statistically similar configurations. This unsupervised task is done using the expectation maximization algorithm. Then, for each of the clusters, a neural network is trained to build this mapping. Clustering body configurations improves the mapping accuracy. Given new visual features, a mapping from each cluster is performed providing a set of possible poses. From this set, the most likely pose is extracted given the learned probability distribution and the visual feature similarity between hypothesis and input. Performance of the system is characterized using a new set of known body postures, showing promising results.
---
paper_title: Learning and recognizing human dynamics in video sequences

paper_content:

This paper describes a probabilistic decomposition of human dynamics at multiple abstractions, and shows how to propagate hypotheses across space, time, and abstraction levels. Recognition in this framework is the succession of very general low level grouping mechanisms to increased specific and learned model based grouping techniques at higher levels. Hard decision thresholds are delayed and resolved by higher level statistical models and temporal context. Low-level primitives are areas of coherent motion found by EM clustering, mid-level categories are simple movements represented by dynamical systems, and high-level complex gestures are represented by Hidden Markov Models as successive phases of ample movements. We show how such a representation can be learned from training data, and apply It to the example of human gait recognition.
---
paper_title: Tracking hybrid 2D-3D human models from multiple views

paper_content:

A novel framework is proposed under which robust matching and tracking of a 3D skeleton model of a human body from multiple views can be performed We propose a method for measuring the ambiguity of 2D measurements provided by each view. The ambiguity measurement is then used for selecting the best view for the most accurate match and tracking. A hybrid 2D-3D representation is chosen for modelling human body poses. The hybrid model is learnt using hierarchical principal component analysis. The CONDENSATION algorithm is used to robustly track and match 3D skeleton models in individual views.
---
paper_title: Tracking a person with 3-D motion by integrating optical flow and depth

paper_content:

This paper describes a method of tracking a person with 3D translation and rotation by integrating optical flow and depth. The target region is first extracted based on the probability of each pixel belonging to the target person. The target state (3D position, posture, motion) is estimated based on the shape and the position of the target region in addition to optical flow and depth. Multiple target states are maintained when the image measurements give rise to ambiguities about the target state. Experimental results with real image sequences show the effectiveness of our method.
---
paper_title: Constraint-conscious smoothing framework for the recovery of 3D articulated motion from image sequences

paper_content:

3D articulated motion is recovered from image sequences by relying on a recursive smoothing framework. In conventional recursive filtering frameworks, the filter may misestimate the state due to degenerated observation. To cope with this problem, we take into account knowledge about the limitations of the state-space. Our novel estimation framework relies on the combination of a smoothing algorithm with a "constraint-conscious" enhanced Kalman filter. The technique is shown to be effective for the recovery of experimental 3D articulated motions, making it a good candidate for marker-less motion capture applications.
---
paper_title: Human action tracking guided by key-frames

paper_content:

The model-based approaches for tracking of human bodies in image sequences can be categorised into two types; fitting model to body frame by frame, and accumulating estimated pose displacements in successive frames after model fitting at the initial frame. The latter has an inherent drawback as accumulation of tracking errors while the one has a great advantage as small computational efforts compared with the former. This paper proposes a new method which can correct the tracking errors by propagation from fitting model to body at a few key-frames. The propagation makes it possible to establish tracking of bodies under occlusion. Capturing the actor's motions in real old movies is presented.
---
paper_title: 3-D model-based tracking of humans in action: a multi-view approach

paper_content:

We present a vision system for the 3-D model-based tracking of unconstrained human movement. Using image sequences acquired simultaneously from multiple views, we recover the 3-D body pose at each time instant without the use of markers. The pose-recovery problem is formulated as a search problem and entails finding the pose parameters of a graphical human model whose synthesized appearance is most similar to the actual appearance of the real human in the multi-view images. The models used for this purpose are acquired from the images. We use a decomposition approach and a best-first technique to search through the high dimensional pose parameter space. A robust variant of chamfer matching is used as a fast similarity measure between synthesized and real edge images. We present initial tracking results from a large new Humans-in-Action (HIA) database containing more than 2500 frames in each of four orthogonal views. They contain subjects involved in a variety of activities, of various degrees of complexity, ranging from the more simple one-person hand waving to the challenging two-person close interaction in the Argentine Tango.
---
paper_title: Tracking people with twists and exponential maps

paper_content:

This paper demonstrates a new visual motion estimation technique that is able to recover high degree-of-freedom articulated human body configurations in complex video sequences. We introduce the use of a novel mathematical technique, the product of exponential maps and twist motions, and its integration into a differential motion estimation. This results in solving simple linear systems, and enables us to recover robustly the kinematic degrees-of-freedom in noise and complex self occluded configurations. We demonstrate this on several image sequences of people doing articulated full body movements, and visualize the results in re-animating an artificial 3D human model. We are also able to recover and re-animate the famous movements of Eadweard Muybridge's motion studies from the last century. To the best of our knowledge, this is the first computer vision based system that is able to process such challenging footage and recover complex motions with such high accuracy.
---
paper_title: Image sequence analysis of real world human motion

paper_content:

Abstract Three fundamental problems of image sequence analysis are investigated for real world human motions. The first one is the modelling of body motion, which is approached by the methods of key frame sequence of stick figures and generalized cone approximation of body parts. The second problem is the correspondence of feature points in the consecutive frames, for which a method called “window code matching” is proposed. The last one is the analysis of occluding regions, which is studied on the bases of occlusion prediction by model and difference images.
---

Title: A Survey of Computer Vision-Based Human Motion Capture

Section 1: INTRODUCTION
Description 1: This section introduces the topic and summarizes the purpose and scope of the survey.

Section 2: A Taxonomy Based on Functionalities
Description 2: This section discusses various taxonomies with a focus on the functionalities within motion capture systems.

Section 3: Assumptions
Description 3: This section outlines the various assumptions regarding movement and appearance that characterize different systems.

Section 4: INITIALIZATION
Description 4: This section dives into the initialization process critical for ensuring accurate system operation at the start.

Section 5: TRACKING
Description 5: This section defines tracking and discusses figure-ground segmentation, representations, and tracking over time.

Section 6: POSE ESTIMATION
Description 6: This section explains the process of estimating the pose configuration of humans and details different approaches like model-free, indirect, and direct model use.

Section 7: RECOGNITION
Description 7: This section covers the recognition of human actions and the paradigms of recognition by reconstruction and direct recognition.

Section 8: DISCUSSION
Description 8: This section evaluates and compares different systems and offers general comments regarding their performance in various application areas.

Section 9: Future Directions
Description 9: This section highlights potential future research directions and improvements in the field of computer vision-based human motion capture.

Section 10: CONCLUSION
Description 10: This section provides a summary of the field's current state and discusses the importance and potential future of human motion capture technology.