A Survey of Applications and Human Motion Recognition with Microsoft Kinect
---
paper_title: FAAST: The Flexible Action and Articulated Skeleton Toolkit

paper_content:

The Flexible Action and Articulated Skeleton Toolkit (FAAST) is middleware to facilitate integration of full-body control with virtual reality applications and video games using OpenNI-compliant depth sensors (currently the PrimeSensor and the Microsoft Kinect). FAAST incorporates a VRPN server for streaming the user's skeleton joints over a network, which provides a convenient interface for custom virtual reality applications and games. This body pose information can be used for goals such as realistically puppeting a virtual avatar or controlling an on-screen mouse cursor. Additionally, the toolkit also provides a configurable input emulator that detects human actions and binds them to virtual mouse and keyboard commands, which are sent to the actively selected window. Thus, FAAST can enable natural interaction for existing off-the-shelf video games that were not explicitly developed to support input from motion sensors. The actions and input bindings are configurable at run-time, allowing the user to customize the controls and sensitivity to adjust for individual body types and preferences. In the future, we plan to substantially expand FAAST's action lexicon, provide support for recording and training custom gestures, and incorporate real-time head tracking using computer vision techniques.
---
paper_title: A Kinect-based system for physical rehabilitation: A pilot study for young adults with motor disabilities

paper_content:

Abstract This study assessed the possibility of rehabilitating two young adults with motor impairments using a Kinect-based system in a public school setting. This study was carried out according to an ABAB sequence in which A represented the baseline and B represented intervention phases. Data showed that the two participants significantly increased their motivation for physical rehabilitation, thus improving exercise performance during the intervention phases. Practical and developmental implications of the findings are discussed.
---
paper_title: Development and evaluation of low cost game-based balance rehabilitation tool using the microsoft kinect sensor

paper_content:

The use of the commercial video games as rehabilitation tools, such as the Nintendo WiiFit, has recently gained much interest in the physical therapy arena. Motion tracking controllers such as the Nintendo Wiimote are not sensitive enough to accurately measure performance in all components of balance. Additionally, users can figure out how to "cheat" inaccurate trackers by performing minimal movement (e.g. wrist twisting a Wiimote instead of a full arm swing). Physical rehabilitation requires accurate and appropriate tracking and feedback of performance. To this end, we are developing applications that leverage recent advances in commercial video game technology to provide full-body control of animated virtual characters. A key component of our approach is the use of newly available low cost depth sensing camera technology that provides markerless full-body tracking on a conventional PC. The aim of this research was to develop and assess an interactive game-based rehabilitation tool for balance training of adults with neurological injury.
---
paper_title: Natural interaction techniques using Kinect

paper_content:

The work presented focuses on the study of interaction techniques in 3-dimensional (3D) and its implementation with the device Kinect. We analyze this device and its operation in order to know their characteristics and limitations. The main objective is to study how to adapt the classical 3D interaction techniques to Kinect. For this, we have made a study of several modes of interaction that are currently using this device and videogames that use this device. We conclude presenting a case study which use Kinect to implement a system of cognitive rehabilitation.
---
paper_title: SmartGlove for upper extremities rehabilitative gaming assessment

paper_content:

This paper presents a quantitative assessment solution for an upper extremities rehabilitative gaming application [1]. This assessment solution consists of a set of stand-alone hardware, including SmartGlove and Kinect, a depth capturing sensor made by Microsoft. SmartGlove is a specially designed motion and finger angle extraction device which is packaged in an easy-to-wear and adjustable manner for a patient with an upper extremity impairment. Sensor data extraction, alignment, and visualization algorithms were designed for integrating hand-mounted sensors data streams into skeleton coordinates captured by the Kinect. This enhanced skeleton information can be summarized and replayed as upper extremity joint coordinate animations which can be used for physical therapists to quantify rehabilitation progress. In addition, to serve as an assessment tool, enhanced skeleton information can be used to extend the capability of the Kinect vision system, such as providing motion capture of the upper extremities, even when the testing subject is out of camera scope or one's upper extremities are occluded by the body.
---
paper_title: Interactive game-based rehabilitation using the Microsoft Kinect

paper_content:

Using video games in rehabilitation settings has the potential to provide patients with fun and motivating exercise tools. Within the Medical VR and MxR groups at the USC Institute for Creative Technologies, we have been leveraging the technology of the Microsoft Kinect 3D depth-sensing camera. Our Kinect-based rehabilitation game “JewelMine” consists of a set of static balance training exercises which encourage the players to reach out of their base of support. We plan to demonstrate a sophisticated post-session analysis tool and several content themes which can be changed dynamically during a therapy session.
---
paper_title: Towards pervasive physical rehabilitation using Microsoft Kinect

paper_content:

The use of Virtual Reality technology for developing tools for rehabilitation has attracted significant interest in the physical therapy arena. This paper presents a comparison of motion tracking performance between the low-cost Microsoft Kinect and the high fidelity OptiTrack optical system. Data is collected on six upper limb motor tasks that have been incorporated into a game-based rehabilitation application. The experiment results show that Kinect can achieve competitive motion tracking performance as OptiTrack and provide “pervasive” accessibility that enables patients to take rehabilitation treatment in clinic and home environment.
---
paper_title: Mixed reality game prototypes for upper body exercise and rehabilitation

paper_content:

This research demonstration consists of an integrated hardware and software platform developed for rapid prototyping of virtual reality-based games for upper body exercise and rehabilitation. The exercise protocol has been adopted from an evidence-based shoulder exercise program for individuals with spinal cord injury. The hardware consists of a custom metal rig that holds a standard wheelchair, six Gametraks attached to elastic exercise bands, a Microsoft Kinect, a laptop and a large screen. A total of 21 prototypes were built using drivers for Kinect, MaxMSP and Unity Pro 3 in order to evaluate game ideas based on deconstruction of the exercise protocol. Future directions include validation of our heuristic design and evaluation model and the development of an exercise suite of point-of-care VR games.
---
paper_title: MotionMA: motion modelling and analysis by demonstration

paper_content:

Particularly in sports or physical rehabilitation, users have to perform body movements in a specific manner for the exercises to be most effective. It remains a challenge for experts to specify how to perform such movements so that an automated system can analyse further performances of it. In a user study with 10 participants we show that experts' explicit estimates do not correspond to their performances. To address this issue we present MotionMA, a system that: (1) automatically extracts a model of movements demonstrated by one user, e.g. a trainer, (2) assesses the performance of other users repeating this movement in real time, and (3) provides real-time feedback on how to improve their performance. We evaluated the system in a second study in which 10 other participants used the system to demonstrate arbitrary movements. Our results demonstrate that MotionMA is able to extract an accurate movement model to spot mistakes and variations in movement execution.
---
paper_title: Personal Rehabilitation Exercise Assistant with Kinect and Dynamic Time Warping

paper_content:

presence of a professional may cause adverse event or lead to secondary injury. In this paper, we describe our development of a Kinect-based system - KHRD using Dynamic Time Warping (DTW) algorithm and fuzzy logic for ensuring home-based rehabilitation. The KHRD allows a patient perform a prescribed exercise with the presence of a professional. The exercise performed will then be recorded as a base for evaluating the patient's rehabilitation exercise at home. The outcomes of the evaluation can be used as a reference for the patient to validate his/her exercise and to prevent adverse events. A summary report of the outcomes may also be uploaded to a cloud setting for physicians to monitor the patient's progress and adjust the prescription. B. Kinect
---
paper_title: Rule based realtime motion assessment for rehabilitation exercises

paper_content:

In this paper, we describe a rule based approach to realtime motion assessment of rehabilitation exercises. We use three types of rules to define each exercise: (1) dynamic rules, with each rule specifying a sequence of monotonic segments of the moving joint or body segment, (2) static rules for stationary joints or body segments, and (3) invariance rules that dictate the requirements of moving joints or body segments. A finite state machine based approach is used in dynamic rule specification and realtime assessment. In addition to the typical advantages of the rule based approach, such as realtime motion assessment with specific feedback, our approach has the following advantages: (1) increased reusability of the defined rules as well as the rule assessment engine facilitated by a set of generic rule elements; (2) increased customizability of the rules for each exercise enabled by the use of a set of generic rule elements and the use of extensible rule encoding method; and (3) increased robustness without relying on expensive statistical algorithms to tolerate motion sensing errors and subtle patient errors.
---
paper_title: A Kinect-based rehabilitation exercise monitoring and guidance system

paper_content:

In this paper, we describe the design and implementation of a Kinect-based system for rehabilitation exercises monitoring and guidance. We choose to use the Unity framework to implement our system because it enables us to use virtual reality techniques to demonstrate detailed movements to the patient, and to facilitate examination of the quality and quantity of the patient sessions by the clinician. The avatar-based rendering of motion also preserves the privacy of the patients, which is essential for healthcare systems. The key contribution of our research is a rule-based approach to realtime exercise quality assessment and feedback. We developed a set of basic rule elements that can be used to express the correctness rules for common rehabilitation exercises.
---
paper_title: Simultaneous categorical and spatio-temporal 3D gestures using Kinect

paper_content:

Recent technological advances have led to an increasing popularity of 3D gesture-based interfaces, in particular in gaming and entertainment consoles. However, unlike 2D gestures, which have been successfully utilized in many multi-touch devices, developing a 3D gesture-based interface is not an easy endeavor. Reasons include the complexity of capturing human movements in 3D and the difficulties associated with recognizing gestures from human motion data. In this work, we target the latter problem by proposing a novel gesture recognition technique for skeletal input data that simultaneously allows for categorical and spatio-temporal gestures. In other words, it recognizes the gesture type and the relative pose within a gesture at the same time. Moreover, our method can learn gestures that are most appropriate for the user from examples. In order to avoid the need for user-specific training, we further propose and evaluate several types of feature representations for human pose data. We argue how our approach can facilitate the development of a customizable 3D gesture-based interface and explore possibilities in order to smoothly integrate the proposed recognition approach into available component-based user interface frameworks. Besides a quantitative evaluation, we present a user study in the scenario of a 3D gesture-based interface for an intra-operative medical image viewer. Our studies support the applicability of our method for developing 3D gesture-based interfaces in practice.
---
paper_title: Controller-free exploration of medical image data: Experiencing the Kinect

paper_content:

In this paper, an open-source system for a controller-free, highly interactive exploration of medical images is presented. By using a Microsoft Xbox Kinect™ as the only input device, the system's user interface allows users to interact at a distance through hand and arm gestures. The paper also details the interaction techniques we have designed specifically for the deviceless exploration of medical imaging data. Since the user interface is touch-free and does not require complex calibration steps, it is suitable for use in operating rooms, where non-sterilizable devices cannot be used.
---
paper_title: Fall detection based on skeleton extraction

paper_content:

This paper presents an improved skeleton extraction from depth video for fall detection based on fast randomized decision forest (RDF) algorithm. Due to the human's body orientation changes dramatically during falling, it reduces the accuracy of tracking. The human's orientation needs to be corrected before the process by RDF. A rotation to correct the orientation is required frame by frame. Experimental results show that with the help of correction our proposed fall detection method could outperform the existing RDF based method.
---
paper_title: Fall Detection from Depth Map Video Sequences

paper_content:

Falls are one of the major risks for seniors living alone at home. Computer vision systems, which do not require to wear sensors, offer a new and promising solution for fall detection. In this work, an occlusion robust method is presented based on two features: human centroid height relative to the ground and body velocity. Indeed, the first feature is an efficient solution to detect falls as the vast majority of falls ends on the ground or near the ground. However, this method can fail if the end of the fall is completely occluded behind furniture. Fortunately, these cases can be managed by using the 3D person velocity computed just before the occlusion.
---
paper_title: 3D Human Motion Analysis to Detect Abnormal Events on Stairs

paper_content:

Falls on the stairs are a common cause of accidental injury among the older adults. Understanding the mechanisms leading to such accidents may improve not only the prevention of falls, but also support independent living among elderly. Thus, a method to automatically detect falls and other abnormal events on stairs is presented and empirically validated. Automatic fall detection will also assist in data collection for environmental design improvements and fall prevention. Real-time 3D joint tracking information, provided by a Microsoft Kinect, is used to estimate the walking speed and to extract a set of features that encode human motion during stairway descent. Supervised learning algorithms, trained on manually labelled training data simulated in a home laboratory, obtained a high detection accuracy rate of ~ 92\% in leave-one-subject-out cross validation. In contrast with previous research, which identified visual tracking of the feet as the best indicator of dangerous activity, 3D motion of the hips is experimentally shown to be the most informative component in detecting abnormal events in the 3D tracking data provided by the Kinect.
---
paper_title: Passive in-home measurement of stride-to-stride gait variability comparing vision and Kinect sensing

paper_content:

We present an analysis of measuring stride-to-stride gait variability passively, in a home setting using two vision based monitoring techniques: anonymized video data from a system of two web-cameras, and depth imagery from a single Microsoft Kinect. Millions of older adults fall every year. The ability to assess the fall risk of elderly individuals is essential to allowing them to continue living safely in independent settings as they age. Studies have shown that measures of stride-to-stride gait variability are predictive of falls in older adults. For this analysis, a set of participants were asked to perform a number of short walks while being monitored by the two vision based systems, along with a marker based Vicon motion capture system for ground truth. Measures of stride-to-stride gait variability were computed using each of the systems and compared against those obtained from the Vicon.
---
paper_title: A viewpoint-independent statistical method for fall detection

paper_content:

The goal of a fall detection system is to automatically detect cases where a human falls and may have been injured. We propose a statistical method based on Kinect depth cameras, that makes a decision based on information about how the human moved during the last few frames. Our method proposes novel features to be used for fall detection, and combines those features using a Bayesian framework. Our experiments explicitly evaluate the ability of our method to use training data collected from one viewpoint, in order to recognize falls from a different viewpoint. We obtain promising results, on a challenging dataset, that we have made public, and that contains, in addition to falls, several similar-looking events such as sitting down, picking up objects from under the bed, or tying shoelaces.
---
paper_title: RGBD-camera based get-up event detection for hospital fall prevention

paper_content:

In this work, we develop a computer vision based fall prevention system for hospital ward application. To prevent potential falls, once the event of patient get up from the bed is automatically detected, nursing staffs are alarmed immediately for assistance. For the detection task, we use a RGBD sensor (Microsoft Kinect). The geometric prior knowledge is exploited by identifying a set of task-specific feature channels, e.g., regions of interest. Extensive motion and shape features from both color and depth image sequences are extracted. Features from multiple modalities and channels are fused via a multiple kernel learning framework for training the event detector. Experimental results demonstrate the high accuracy and efficiency achieved by the proposed system.
---
paper_title: Fall Detection from Depth Map Video Sequences

paper_content:

Falls are one of the major risks for seniors living alone at home. Computer vision systems, which do not require to wear sensors, offer a new and promising solution for fall detection. In this work, an occlusion robust method is presented based on two features: human centroid height relative to the ground and body velocity. Indeed, the first feature is an efficient solution to detect falls as the vast majority of falls ends on the ground or near the ground. However, this method can fail if the end of the fall is completely occluded behind furniture. Fortunately, these cases can be managed by using the 3D person velocity computed just before the occlusion.
---
paper_title: 3D Human Motion Analysis to Detect Abnormal Events on Stairs

paper_content:

Falls on the stairs are a common cause of accidental injury among the older adults. Understanding the mechanisms leading to such accidents may improve not only the prevention of falls, but also support independent living among elderly. Thus, a method to automatically detect falls and other abnormal events on stairs is presented and empirically validated. Automatic fall detection will also assist in data collection for environmental design improvements and fall prevention. Real-time 3D joint tracking information, provided by a Microsoft Kinect, is used to estimate the walking speed and to extract a set of features that encode human motion during stairway descent. Supervised learning algorithms, trained on manually labelled training data simulated in a home laboratory, obtained a high detection accuracy rate of ~ 92\% in leave-one-subject-out cross validation. In contrast with previous research, which identified visual tracking of the feet as the best indicator of dangerous activity, 3D motion of the hips is experimentally shown to be the most informative component in detecting abnormal events in the 3D tracking data provided by the Kinect.
---
paper_title: RGBD-camera based get-up event detection for hospital fall prevention

paper_content:

In this work, we develop a computer vision based fall prevention system for hospital ward application. To prevent potential falls, once the event of patient get up from the bed is automatically detected, nursing staffs are alarmed immediately for assistance. For the detection task, we use a RGBD sensor (Microsoft Kinect). The geometric prior knowledge is exploited by identifying a set of task-specific feature channels, e.g., regions of interest. Extensive motion and shape features from both color and depth image sequences are extracted. Features from multiple modalities and channels are fused via a multiple kernel learning framework for training the event detector. Experimental results demonstrate the high accuracy and efficiency achieved by the proposed system.
---
paper_title: Enhancing realism of mixed reality applications through real-time depth-imaging devices in X3D

paper_content:

Until recently, depth sensing cameras have been used almost exclusively in research due to the high costs of such specialized equipment. With the introduction of the Microsoft Kinect device, realtime depth imaging is now available for the ordinary developer at low expenses and so far it has been received with great interest from both the research and hobby developer community. The underlying OpenNI framework not only allows to extract the depth image from the camera, but also provides tracking information of gestures or user skeletons. In this paper, we present a framework to include depth sensing devices into X3D in order to enhance visual fidelity of X3D Mixed Reality applications by introducing some extensions for advanced rendering techniques. We furthermore outline how to calibrate depth and image data in a meaningful way through calibration for devices that do not already come with precalibrated sensors, as well as a discussion of some of the OpenNI functionality that X3D can benefit from in the future.
---
paper_title: Research on Skeleton Animation Motion Data Based on Kinect

paper_content:

Skeleton animation motion data is the connection of motion capture and computer skeleton animation technologies. This paper presents a method which creates standard motion data file in real time. First, get original coordinate data of human skeleton joints by Kinect, Then, calculate the original coordinate data to get rotation angles and save them as the standard BVH motion capture file, Finally, use the BVH file to drive 3D human model. the experiment results prove that this motion capture system can restore 3D skeleton motion of the captured object correctly. Comparing with the other motion capture systems, it largely reduces the funding of implementing human skeleton animation motion data capture and common motion data storage.
---
paper_title: Perception analysis of motion contributing to individuality using Kinect sensor

paper_content:

This paper investigates that human tend to percept of humanness and individuality to what features of human motion. We analyze from experiments based on test subjects how recognize of individuality by comparing a developed own self motion with other motion made by motion capture using Kinect sensor. The results showed except for the subjects could not perform the movement extremely distinctive, many subjects to recognize the movement of their own. In addition, the respondents also recognized with a video camera an objective of its own motion, the subjects of some of their own and do not match the image of the inside of me. It is difficult to percept humanness based on less-accurate expression of human motion.
---
paper_title: A Wizard-of-Oz elicitation study examining child-defined gestures with a whole-body interface

paper_content:

This paper explores the use of a guessability study to examine child-defined gestures with Kinect. Applying a Wizard-of-Oz approach, gestures were elicited from six children (age 3--8) through a series of 22 task stimuli including object manipulation, navigation-based tasks, and spatial interaction. Gestures were video recorded, transcribed, and coded by three researchers employing an inductive, qualitative method of analysis. Five themes emerged from the data: (1) the influence of 2D touchscreens on children's interactions in 3D, (2) the role of contextual cues in designing a stimuli set, (3) individual preferences for dominant styles of interaction, (4) different approaches children employ to simulate the same object path, and (5) and allocentric versus egocentric approaches for manipulating objects on screen. While we did not achieve strong consensus among all of the gestures produced by children in our study, our results provide a basis for further refinement of the stimulus set and methodology used for future work examining child-defined gestures for whole-body interfaces.
---
paper_title: Interaction System of Treadmill Games based on depth maps and CAM-Shift

paper_content:

Because of the non-contact and substantivity, recognition from images have become a hotspot in somatic technology. Especially, recognition with non-markers lets human body become remote controller without any accessories and attracts widespread attention. This paper chose CAM-Shift algorithm to track bare hands for its calculate speed and stability. This article introduced depth information to traditional CAM-Shift and developed and extended the algorithm in foreground segmentation, and auto-initialization. This paper mainly segmented depth map to get the mask off code of the foreground, and made use of the 3D information of the depth maps to initialize the searching window. Then, the paper built up Interaction System of Treadmill Games based on depth maps and CAM-Shift. The system realized the man-machine interaction with non-contact and no-markers by getting player's depth maps and color images with a attachment of Xbox — Kinect from Microsoft, 3D tracking the player's bare hands and analyzing and recognizing the movements of the hands. Then the experiments testified this system satisfies the real time request, and the reaction is acute and correct.
---
paper_title: HoloDesk: direct 3d interactions with a situated see-through display

paper_content:

HoloDesk is an interactive system combining an optical see through display and Kinect camera to create the illusion that users are directly interacting with 3D graphics. A virtual image of a 3D scene is rendered through a half silvered mirror and spatially aligned with the real-world for the viewer. Users easily reach into an interaction volume displaying the virtual image. This allows the user to literally get their hands into the virtual display and to directly interact with an spatially aligned 3D virtual world, without the need for any specialized head-worn hardware or input device. We introduce a new technique for interpreting raw Kinect data to approximate and track rigid (e.g., books, cups) and non-rigid (e.g., hands, paper) physical objects and support a variety of physics-inspired interactions between virtual and real. In particular the algorithm models natural human grasping of virtual objects with more fidelity than previously demonstrated. A qualitative study highlights rich emergent 3D interactions, using hands and real-world objects. The implementation of HoloDesk is described in full, and example application scenarios explored. Finally, HoloDesk is quantitatively evaluated in a 3D target acquisition task, comparing the system with indirect and glasses-based variants.
---
paper_title: Code Space: Touch + Air Gesture Hybrid Interactions for Supporting Developer Meetings

paper_content:

We present Code Space, a system that contributes touch + air gesture hybrid interactions to support co-located, small group developer meetings by democratizing access, control, and sharing of information across multiple personal devices and public displays. Our system uses a combination of a shared multi-touch screen, mobile touch devices, and Microsoft Kinect sensors. We describe cross-device interactions, which use a combination of in-air pointing for social disclosure of commands, targeting and mode setting, combined with touch for command execution and precise gestures. In a formative study, professional developers were positive about the interaction design, and most felt that pointing with hands or devices and forming hand postures are socially acceptable. Users also felt that the techniques adequately disclosed who was interacting and that existing social protocols would help to dictate most permissions, but also felt that our lightweight permission feature helped presenters manage incoming content.
---
paper_title: Empirical study of a vision-based depth-sensitive human-computer interaction system

paper_content:

This paper proposes the results of a user study on vision-based depth-sensitive input system for performing typical desktop tasks through arm gestures. We have developed a vision-based HCI prototype to be used for our comprehensive usability study. Using the Kinect 3D camera and OpenNI software library we implemented our system with high stability and efficiency by decreasing the ambient disturbing factors such as noise or light condition dependency. In our prototype, we designed a capable algorithm using NITE toolkit to recognize arm gestures. Finally, through a comprehensive user experiment we compared our natural arm gestures to the conventional input devices (mouse/keyboard), for simple and complicated tasks, and in two different situations (small and big-screen displays) for precision, efficiency, ease-of-use, pleasantness, fatigue, naturalness, and overall satisfaction to verify the following hypothesis: on a WIMP user interface, the gesture-based input is superior to mouse/keyboard when using big-screen. Our empirical investigation also proves that gestures are more natural and pleasant to be used than mouse/keyboard. However, arm gestures can cause more fatigue than mouse.
---
paper_title: Kinected browser: depth camera interaction for the web

paper_content:

Interest in and development of gesture interfaces has recently exploded, fueled in part by the release of Microsoft Corporation's Kinect, a low-cost, consumer-packaged depth camera with integrated skeleton tracking. Depth-camera-based gestures can facilitate interaction with the Web on keyboard-and-mouse-free and/or multi-user technologies, such as large display walls or TV sets. We present a toolkit for bringing such gesture affordances into modern Web browsers using existing Web programming methods. Our framework is designed to enable Web programmers to incrementally add this capability with minimum effort by leveraging Web standard DOM structures and event models. We describe our framework's design and architecture, and illustrate its usability and versatility.
---
paper_title: Kinect based 3D object manipulation on a desktop display

paper_content:

Gesture-based controllers such as the Microsoft Kinect are low cost devices that allow a user to interact with complex, three-dimensional simulations using an interface argued to be more natural than game controllers, joy sticks, or a mouse and keyboard. This paper presents a controlled experimental evaluation of the use of Microsoft Kinect to support a 3D object manipulation task. Users were asked to match the orientation of objects with a manipulation interface that displayed either a self-avatar hand and arm or a sphere, both corresponding to users' arm gestures and wrist rotation. Our results show that while there was no overall difference in performance between the self-avatar and sphere visual display conditions, there were clear differences in the two visual display conditions as a function of gender and video-game experience.
---
paper_title: Automatic Camera Control for Tracking a Presenter during a Talk

paper_content:

Today, talks, presentations, and lectures are often captured on video to give a broad audience the possibility to (re-)access the content. As presenters are often moving around during a talk it is necessary to guide recording cameras. We present an automatic solution for user tracking and camera control. It uses a depth camera for user tracking, and a scalable networking architecture based on publish/subscribe messaging for controlling multiple video cameras. Furthermore, we present our experiences with the system during actual lectures at an university.
---
paper_title: Wiimote and Kinect: gestural user interfaces add a natural third dimension to HCI

paper_content:

The recent diffusion of advanced controllers, initially designed for the home game console, has been rapidly followed by the release of proprietary or third part PC drivers and SDKs suitable for implementing new forms of 3D user interfaces based on gestures. Exploiting the devices currently available on the game market, it is now possible to enrich, with low cost motion capture, the user interaction with desktop computers by building new forms of natural interfaces and new action metaphors that add the third dimension as well as a physical extension to interaction with users. This paper presents two systems specifically designed for 3D gestural user interaction on 3D geographical maps. The proposed applications rely on two consumer technologies both capable of motion tracking: the Nintendo Wii and the Microsoft Kinect devices. The work also evaluates, in terms of subjective usability and perceived sense of Presence and Immersion, the effects on users of the two different controllers and of the 3D navigation metaphors adopted. Results are really encouraging and reveal that, users feel deeply immerse in the 3D dynamic experience, the gestural interfaces quickly bring the interaction from novice to expert style and enrich the synthetic nature of the explored environment exploiting user physicality.
---
paper_title: A Study on the control Method of 3-Dimensional Space Application using KINECT System

paper_content:

Summary This paper suggests how to control 3D application using KINECT system introduced by Microsoft. The conventional system uses a mouse and a touch sensor to control 3D application. Recently, the size of the display became so large that the existing methods are more difficult to control the application. To solve this problem, I propose the method which controls the application intuitively by using the distance information and the joints’ location information.
---
paper_title: Shape your body: control a virtual silhouette using body motion

paper_content:

In this paper we propose to use our body as a puppetry controller, giving life to a virtual silhouette through acting. A framework was deployed based on Microsoft Kinect using OpenNI and Unity to animate in real-time a silhouette. This was used to perform a set of experiments related to the user's interaction with human and non-human like puppets. We believe that a performance-driven silhouette can be just as expressive as a traditional shadow puppet with a high degree of freedom, making use of our entire body as an input. We describe our solution that allows real-time interactive control of virtual shadow puppets for performance animation based on body motion. We show through our experiment, performed by non-expert artists, that using our body to control puppets is like mixing the performance of an actor with the manipulation of a puppeteer.
---
paper_title: Teaching natural user interaction using OpenNI and the Microsoft Kinect sensor

paper_content:

The launch of the Microsoft Kinect for Xbox (a real-time 3D imaging device) and supporting libraries spurred the development of various applications including those with natural user interfaces. We propose that using Kinect offers opportunities for novel approaches to classroom instruction on natural user interaction. A number of development frameworks has come up that can be used to facilitate this instruction. We evaluate the current state of this technology and present an overview of some of its development frameworks. We then present examples of how Kinect-assisted instruction can be used to achieve some of the learning outcomes in Human Computer Interaction (HCI) courses as outlined in IT2008. We have verified that OpenNI, with its accompanying libraries, can be used for these activities in multi-platform learning environments.
---
paper_title: Music conductor gesture recognized interactive music generation system

paper_content:

In the research of interactive music generation, we propose a music generation method, that the computer generates the music automatically, and then the music will be arranged under the human music conductor's gestures, before it outputs to us. In this research, the generated music is processed from chaotic sound, which is generated from the network of chaotic elements in realtime. The music conductor's hand motions are detected by Microsoft Kinect in this system. Music theories are embedded in the algorithm, as a result, the generated music will be richer. Furthermore, we constructed the music generation system and performed the experiment for generating the music composed by human beings.
---
paper_title: MotionDraw: a tool for enhancing art and performance using kinect

paper_content:

Contemporary staged performances frequently utilize advanced lighting and projection techniques. The design and creation of these stage effects are rarely accessible to the actual performers and must be designed by professional lighting designers or highly-paid programmers. With MotionDraw we want to create an affordable system that is easily controlled and manipulated by performers. With intuitive gestures, non-specialized users can control the MotionDraw visual library and interact with the captured visual record of their own movements. Possible uses for our system grew out of research with dancers and performers, and the current technical implementation sets a framework for including additional visual libraries and capabilities.
---
paper_title: Full-body imitation of human motions with kinect and heterogeneous kinematic structure of humanoid robot

paper_content:

In this work, we propose a system that has the ability to reproduce imitated motions of human during continuous and online observation with a humanoid robot. In order to achieve this goal, the problems for imitation have to be solved. In this paper, we treat two main issues. One is mapping between different kinematic structures and the other is computing humanoid body pose that satisfies the static stability generated from the human motion obtained by visual motion capture of the humanoid. The experimental results based on Webots simulation and subsequent execution by a Darwin-OP humanoid robot show the validity of the proposed system in this paper.
---
paper_title: Study on the use of Microsoft Kinect for robotics applications

paper_content:

The Microsoft X-Box Kinect Sensor is a revolutionary new depth camera that is used in the gaming industry to capture motions of people and players efficiently using the technology of an RGB camera and infrared camera to differentiate depth. In the Microsoft X-Box, Kinect was used to sense 3D perception of human's motions. It can also be used for robotic applications, precisely for indoor navigation through the process of reverse engineering. Certain software packages were made available and are open source from “LibFreenect” for Linux machines, Microsoft's Kinect SDK using the Kinect namespace on Visual Studio 2010 Express (C++, C# or Visual Basic), and Google's released “Robotic Operating System (ROS)”. In order to claim that this sensor is capable of taking on such a task, we must be able to investigate thoroughly all factors that contribute to this and at the same time we must be able to understand its limitations to be applied and integrated properly with certain types of robots for accomplishing our purpose of achieving successful indoor navigation using proper algorithms. In this paper, the results from testing the Kinect sensor on an autonomous ground vehicle was given.
---
paper_title: Real-time dynamic gesture recognition system based on depth perception for robot navigation

paper_content:

Natural human robot interaction based on the dynamic hand gesture is becoming a popular research topic in the past few years. The traditional dynamic gesture recognition methods are usually restricted by the factors of illumination condition, varying color and cluttered background. The recognition performance can be improved by using the hand-wearing devices but this is not a natural and barrier-free interaction. To overcome these shortcomings, the depth perception algorithm based on the Kinect depth sensor is introduced to carry out 3D hand tracking. We propose a novel start/end point detection method for segmenting the 3D hand gesture from the hand motion trajectory. Then Hidden Markov Models (HMMs) are implemented to model and classify the hand gesture sequences and the recognized gestures are converted to control commands for the interaction with the robot. Seven different hand gestures performed by two hands can sufficiently navigate the robot. Experiments show that the proposed dynamic hand gesture interaction system can work effectively in the complex environment and in real-time with an average recognition rate of 98.4%. And further experiments for the robot navigation also verify the robustness of our system.
---
paper_title: Design and implementation of human-robot interactive demonstration system based on Kinect

paper_content:

With the development of technology, humanoid robots gradually enter our life, not only for education, but helping people with housework and many other tasks, some of which are inevitable for human. But only few people know how to control and interact with a humanoid robot, which hinders the development of humanoid robot. So a human-robot interactive demonstration system is designed to help non-expert users to control the humanoid robot, Aldebaran humanoid robot Nao in this case. Users just need to use the most natural body gestures to interact with the robot. Microsoft Kinect is applied in this system to recognize different body gestures and generate visual Human-Robot interaction interface, then the controlling signals of different body gesture modules are sent to Nao through wifi, which can stimulate Nao to complete tasks. This kind of system aims to enrich the interactive way between human and robots and help non-expert users to control the robot freely, making Human-Robot interaction much easier.
---
paper_title: Recognition of Human Motions for Imitation and Control of a Humanoid Robot

paper_content:

Most of the works in the field of teleoperation of humanoid robots do not use the whole-body of the robot to perform some task. This work treats the teleoperation of a humanoid robot through the recognition of human motions by using a natural interface mechanism. We propose the use of simple mathematical techniques to allow, in a satisfactory level, the user order in real-time a robot to mimic a person, to walk, to manipulate the environment with its hands and to perform some predetermined behaviors. Also, we provide an appropriate discussion around the findings of this research. The results of the experiments show a satisfactory performance for the system developed.
---
paper_title: A real-time human imitation system

paper_content:

Robot imitation is a useful tool in humanoid robot research, it provides a natural way of teaching a complex humanoid robot to accomplish human-like behaviors. However, it is never an easy task to build such a system robustly due to the high degree of freedom (DOF) in human motion and difficulty in humanoid control. In this paper, we present a system built up by the Microsoft Kinect box and the Aldebaran NAO Humanoid robot, which can mimic full body human motion in real time. By applying inverse kinematics through an optimization process, the human motion was split up into critical frames and represented by a list of robot joint angles. Then a second order continuous trajectory is formed to drive each joint toward the target angle, followed by balance control and self-collision avoidance system. The tests show that the system is robust and flexible enough to imitate various human motions.
---
paper_title: Altitude control of a quadrotor helicopter using depth map from Microsoft Kinect sensor

paper_content:

Reliable depth estimation is a cornerstone of many autonomous robotic control systems. The Microsoft Kinect is a new, low cost, commodity game controller peripheral that calculates a depth map of the environment with good accuracy and high rate. In this paper we calibrate the Kinect depth and image sensors and then use the depth map to control the altitude of a quadrotor helicopter. This paper presents the first results of using this sensor in a real-time robotics control application.
---
paper_title: In Situ Motion Capture of Speed Skating: Escaping the Treadmill

paper_content:

The advent of the Kinect depth imager has opened the door to motion capture applications that would have been much more costly with previous technologies. In part, the Kinect achieves this by focusing on a very specific application domain, thus narrowing the requirement for the motion capture system. Specifically, Kinect motion capture works best within a small physical space while the camera is stationary. We seek to extend Kinect motion capture for use in athletic training - speed skating in particular - by placing the Kinect on a mobile, robotic platform to capture motion in situ. Athletes move over large distances, so the mobile platform addresses the limited viewing area of the Kinect. As the platform moves, we must also account for the now dynamic background against which the athlete performs. The result is a novel, visually-guided robotic platform that follows athletes, allowing us to capture motion and images that would not be possible with a treadmill. We describe the system in detail and give examples of the system capturing the motion of a speed skater at typical training speeds.
---
paper_title: Review: Feature extraction in Brazilian Sign Language Recognition based on phonological structure and using RGB-D sensors

paper_content:

In contrast to speech recognition, whose speech features have been extensively explored in the research literature, feature extraction in Sign Language Recognition (SLR) is still a very challenging problem. In this paper we present a methodology for feature extraction in Brazilian Sign Language (BSL, or LIBRAS in Portuguese) that explores the phonological structure of the language and relies on RGB-D sensor for obtaining intensity, position and depth data. From the RGB-D images we obtain seven vision-based features. Each feature is related to one, two or three structural elements in BSL. We investigate this relation between extracted features and structural elements based on shape, movement and position of the hands. Finally we employ Support Vector Machines (SVM) to classify signs based on these features and linguistic elements. The experiments show that the attributes of these elements can be successfully recognized in terms of the features obtained from the RGB-D images, with accuracy results individually above 80% on average. The proposed feature extraction methodology and the decomposition of the signs into their phonological structure is a promising method to help expert systems designed for SLR.
---
paper_title: Kinect-based Taiwanese sign-language recognition system

paper_content:

Gesture-recognition is an important component for many intelligent human---computer interaction applications. For example, a realtime sign-language recognition system would detect and interpret hand gestures. Many vision-based sign-language recognition methods have been proposed over the years with mix results of usability. Some system are limited to recognize only a few gestures, while others require the use of 3D camera to provides depth information to improve recognition accuracy. In this paper, a Kinect-based Taiwanese sign-language recognition system is proposed. Three main features are extracted from the signing gestures, namely hand positions, hand signing direction, and hand shapes. The hand positions are readily available through the input sensor. The signing direction is determined using HMM on trajectory of the hand movement, and a SVM is trained and used to recognize the hand shapes. Experimental results show that the proposed system achieved an 85.14 % recognition rate.
---
paper_title: Combining features for Chinese sign language recognition with Kinect

paper_content:

In this paper, we propose a novel three-dimensional combining features method for sign language recognition. Based on the Kinect depth data and the skeleton joints data, we acquire the 3D trajectories of right hand, right wrist and right elbow. To construct feature vector, the paper uses combining location and spherical coordinate feature representation. The proposed approach utilizes the feature representation in spherical coordinate system effectively depicting the kinematic connectivity among hand, wrist and elbow for recognition. Meanwhile, 3D trajectory data acquired from Kinect avoid the interference of the illumination change and cluttered background. In experiments with a dataset of 20 gestures from Chinese sign language, the Extreme Learning Machine(ELM) is tested, compared with Support Vector Machine( SVM), the superior recognition performance is verified.
---
paper_title: A Kinect based sign language recognition system using spatio-temporal features

paper_content:

This paper presents a sign language recognition system that uses spatio-temporal features on RGB video images and depth maps for dynamic gestures of Turkish Sign Language. Proposed system uses motion differences and accumulation approach for temporal gesture analysis. Motion accumulation method, which is an effective method for temporal domain analysis of gestures, produces an accumulated motion image by combining differences of successive video frames. Then, 2D Discrete Cosine Transform (DCT) is applied to accumulated motion images and temporal domain features transformed into spatial domain. These processes are performed on both RGB images and depth maps separately. DCT coefficients that represent sign gestures are picked up via zigzag scanning and feature vectors are generated. In order to recognize sign gestures, K-Nearest Neighbor classifier with Manhattan distance is performed. Performance of the proposed sign language recognition system is evaluated on a sign database that contains 1002 isolated dynamic signs belongs to 111 words of Turkish Sign Language (TSL) in three different categories. Proposed sign language recognition system has promising success rates.
---
paper_title: An Isolated Sign Language Recognition System Using RGB-D Sensor with Sparse Coding

paper_content:

An isolated sign language recognition system is presented in this paper. A RGB-D sensor, Microsoft Kinect, is used for obtaining color stream and skeleton points from the depth stream. For a particular sign we extract a representative feature vector composed by hand trajectories and hand shapes. A sparse dictionary learning algorithm, Label Consistent K-SVD (LC-KSVD), is applied to obtain a discriminative dictionary. Based on that, we further develop a new classification approach to get better result. Our system is evaluated on 34 isolated Chinese sign words including one-handed signs and two-handed signs. Experimental results show the proposed system gets high recognition accuracy, of the reported 96.75%, and obtain an average accuracy of 92.36% for signer independent recognition.
---
paper_title: A real-time system to recognize static gestures of Brazilian sign language (libras) alphabet using Kinect

paper_content:

In this paper we present a system - called Gesture User Interface (GestureUI) - to recognize static gestures of the Brazilian Sign Language (Libras) in real-time out of a continuous video stream using the depth information captured by a Kinect controller. We focus on handling small sets of gestures (A, E, I, O, U) and (B, C, F, L, V), processing them in two steps: Segmentation and Classification. For the Segmentation we propose a Virtual Wall and Libras-specific heuristics to improve the hand tracking. For the classification we use a Multi-Layer Perceptron trained by the system and present an arm cutting algorithm that improved the recognition rate from 67.4% and 75.4% to 100% for both gesture sets. Finally, we evaluated the processing performance of the overall system and proof that it is able to process frames at 62.5Hz with an Intel i7 processor, which is more than twice as fast as the Kinect frame capturing rate.
---
paper_title: Dynamic time wrapping based gesture recognition

paper_content:

Sign language provides hearing and speech impaired people with an interface to communicate with society. Unfortunately most people do not understand sign language. For this, image processing and pattern recognition can provide with a vital tool to detect and translate sign language into vocal language. This work presents a method for detecting, understanding and translating sign language gestures to vocal language. Microsoft Kinect is the primary tool used to capture video stream of the user. This is achieved by getting skeleton frame from Kinect and then extracting joints of interest. The data obtained are normalized and a linked list of skeleton frame is maintained. The proposed method is capable of successfully detecting all gestures that do not involve finger movements. The proposed system has an accuracy of 91%.
---
paper_title: Audio-visual speech recognition using depth information from the Kinect in noisy video conditions

paper_content:

In this paper we build on our recent work, where we successfully incorporated facial depth data of a speaker captured by the Microsoft Kinect device, as a third data stream in an audio-visual automatic speech recognizer. In particular, we focus our interest on whether the depth stream provides sufficient speech information that can improve system robustness to noisy audio-visual conditions, thus studying system operation beyond the traditional scenarios, where noise is applied to the audio signal alone. For this purpose, we consider four realistic visual modality degradations at various noise levels, and we conduct small-vocabulary recognition experiments on an appropriate, previously collected, audiovisual database. Our results demonstrate improved system performance due to the depth modality, as well as considerable accuracy increase, when using both the visual and depth modalities over audio only speech recognition.
---
paper_title: Sign Language Fingerspelling Classification from Depth and Color Images using a Deep Belief Network

paper_content:

Automatic sign language recognition is an open problem that has received a lot of attention recently, not only because of its usefulness to signers, but also due to the numerous applications a sign classifier can have. In this article, we present a new feature extraction technique for hand pose recognition using depth and intensity images captured from a Microsoft Kinect sensor. We applied our technique to American Sign Language finger spelling classification using a Deep Belief Network, for which our feature extraction technique is tailored. We evaluated our results on a multi-user data set with two scenarios: one with all known users and one with an unseen user. We achieved 99% recall and precision on the first, and 77% recall and 79% precision on the second. Our method is also capable of real-time sign classification and is adaptive to any environment or lightning intensity.
---
paper_title: Spelling it out: Real-time ASL fingerspelling recognition

paper_content:

This article presents an interactive hand shape recognition user interface for American Sign Language (ASL) finger-spelling. The system makes use of a Microsoft Kinect device to collect appearance and depth images, and of the OpenNI+NITE framework for hand detection and tracking. Hand-shapes corresponding to letters of the alphabet are characterized using appearance and depth images and classified using random forests. We compare classification using appearance and depth images, and show a combination of both lead to best results, and validate on a dataset of four different users. This hand shape detection works in real-time and is integrated in an interactive user interface allowing the signer to select between ambiguous detections and integrated with an English dictionary for efficient writing.
---
paper_title: Gesture recognition using kinect for sign language translation

paper_content:

Sign Language is a widely used method of communication among the community of deaf-mute people. It contains some series of body gestures, which enables a person to interact without the need of spoken words. Although the use of sign language is very popular among the deaf-mute people but the other communities don't even try to learn it, this creates a gulf of communication and hence becomes a cause of the isolation of physically impaired people. The problem creates a requirement of a system which can facilitate a way of communication between these two communities. This paper presents a novel method for identification of an isolated sign language gesture using Microsoft Kinect. This paper presents the way of extracting some highly robust features from the depth image provided by Kinect and to use them in creating a robust and accurate gesture recognition system, for the purpose of sign language translation. Apart from translation, the proposed system also opens the doors of endless applications in the field of Human Computer Interaction. The proposed algorithm helps in translating a sign language gesture performed by a user, which in-turn can be used as an input for different types of applications.
---
paper_title: American sign language recognition with the kinect

paper_content:

We investigate the potential of the Kinect depth-mapping camera for sign language recognition and verification for educational games for deaf children. We compare a prototype Kinect-based system to our current CopyCat system which uses colored gloves and embedded accelerometers to track children's hand movements. If successful, a Kinect-based approach could improve interactivity, user comfort, system robustness, system sustainability, cost, and ease of deployment. We collected a total of 1000 American Sign Language (ASL) phrases across both systems. On adult data, the Kinect system resulted in 51.5% and 76.12% sentence verification rates when the users were seated and standing respectively. These rates are comparable to the 74.82% verification rate when using the current(seated) CopyCat system. While the Kinect computer vision system requires more tuning for seated use, the results suggest that the Kinect may be a viable option for sign verification.
---
paper_title: Combining depth image and skeleton data from Kinect for recognizing words in the sign system for Indonesian language (SIBI [Sistem Isyarat Bahasa Indonesia])

paper_content:

The Sign System for Indonesian Language (SIBI) is a rather complex sign language. It has four components that distinguish the meaning of the sign language and it follows the syntax and the grammar of the Indonesian language. This paper proposes a model for recognizing the SIBI words by using Microsoft Kinect as the input sensor. This model is a part of automatic translation from SIBI to text. The features for each word are extracted from skeleton and color-depth data produced by Kinect. Skeleton data features indicate the angle between human joints and Cartesian axes. Color images are transformed to gray-scale and their features are extracted by using Discrete Cosine Transform (DCT) with Cross Correlation (CC) operation. The image's depth features are extracted by running MATLAB regionprops function to get its region properties. The Generalized Learning Vector Quantization (GLVQ) and Random Forest (RF) training algorithm from WEKA data mining tools are used as the classifier of the model. Several experiments with different scenarios have shown that the highest accuracy (96,67%) is obtained by using 30 frames for skeleton combined with 20 frames for region properties image classified by Random Forest.
---
paper_title: Hand gesture recognition using color and depth images enhanced with hand angular pose data

paper_content:

In this paper we propose a hand gesture recognition system that relies on color and depth images, and on a small pose sensor on the human palm. Monocular and stereo vision systems have been used for human pose and gesture recognition, but with limited scope due to limitations on texture, illumination, etc. New RGB-Depth sensors, that reply on projected light such as the Microsoft Kinect, have overcome many of those limitations. However, the point clouds for hand gestures are still in many cases noisy and partially occluded, and hand gesture recognition is not trivial. Hand gesture recognition is much more complex than full body motion, since we can have the hands in any orientation and can not assume a standing body on a ground plane. In this work we propose to add a tiny pose sensor to the human palm, with a minute accelerometer and magnetometer that combined provide 3D angular pose, to reduce the search space and have a robust and computationally light recognition method. Starting with the full depth image point cloud, segmentation can be performed by taking into account the relative depth and hand orientation, as well as skin color. Identification is then performed by matching 3D voxel occupancy against a gesture template database. Preliminary results are presented for the recognition of Portuguese Sign Language alphabet, showing the validity of the approach.
---
paper_title: Kinect sensing of shopping related actions

paper_content:

Surveillance systems in shopping malls or supermarkets are usually used for detecting abnormal behavior. We used the distributed video cameras system to design digital shopping assistants which assess the behavior of customers while shopping, detect when they need assistance, and offer their support in case there is a selling opportunity. In this paper we propose a system for analyzing human behavior patterns related to products interaction, such as browse through a set of products, examine, pick products, try on, interact with the shopping cart, and look for support by waiving one hand. We used the Kinect sensor to detect the silhouettes of people and extracted discriminative features for basic action detection. Next we analyzed different classification methods, statistical and also spatio-temporal ones, which capture relations between frames, features, and basic actions. By employing feature level fusion of appearance and movement information we obtained an accuracy of 80% for the mentioned six basic actions.
---
paper_title: Magic Mirror: A virtual handbag shopping system

paper_content:

We present an augmented reality system based on Kinect for on-line handbag shopping. The users can virtually try on different handbags on a TV screen at home. They can interact with the virtual handbags naturally, such as sliding a handbag to different positions on their arms and rotating a handbag to see it from different angles. The users can also see how the handbags fit them in different virtual environments other than the current real background. We describe the technical details for building such a system and demonstrate the experimental results.
---
paper_title: A real-time ergonomic monitoring system using the Microsoft Kinect

paper_content:

Laborers in factories all across the world perform physically intensive tasks daily. With every lift they put themselves at risk of injury. Many still-frame modeling systems exist that can assess the different stresses and strains on the laborers body given his or her position. These models are only usable by experts, and do not allow for real-time alerts. In 1995, companies in the United States lost $50 billion due to injured employee absences and compensation settlements. Companies are not only eager to reduce their overhead costs, but also aim to better society by offering more robust worker safety practices. The focus of this project was to design a system that can be used in a training environment. Our system is used to teach employees if their current lifting and carrying methods can be detrimental to their health. Our system is designed to be used for longstanding employees as well as new hires. This project's primary requirement was to implement a motion sensing device to aid in the analysis of ergonomics in an industrial environment. To do this we proposed to make use of Microsoft Kinect© sensors. The Kinect© is able to provide skeletal tracking at 30 frames/second for two individuals in the field of view. To develop the system we selected the Microsoft software development kit (SDK) from a large variety of alternative professional and open source SDKs because of a variety of desirable features. A static ergonomic model was integrated with the Kinect© software. Multiple other software packages were assessed for compatibility with the Kinect© in an effort to enhance the Kinects'O ability to recognize objects and humans. After development was complete the system was tested by analyzing our system's output using different skeletal lift positions to compare to the real results. Our system provides real-time ergonomic analysis of lifts performed by humans. This system lacks the ability to recognize specific individuals and objects necessary to customize the system to adequately evaluate a lift, and has not been tested in a factory environment. In the future we hope to implement a dynamic ergonomic model so that it can recognize whole movements or gestures which lead to injury, rather than recognizing a single position. Our system successfully outputs a number for the recommended weight limit as well as other methods to measure the strain on a worker's skeleton. In a training environment the system will help individuals correct the problems with their lifting motions.
---
paper_title: Real-Time, Full 3-D Reconstruction of Moving Foreground Objects From Multiple Consumer Depth Cameras

paper_content:

The problem of robust, realistic and especially fast 3-D reconstruction of objects, although extensively studied, is still a challenging research task. Most of the state-of-the-art approaches that target real-time applications, such as immersive reality, address mainly the problem of synthesizing intermediate views for given view-points, rather than generating a single complete 3-D surface. In this paper, we present a multiple-Kinect capturing system and a novel methodology for the creation of accurate, realistic, full 3-D reconstructions of moving foreground objects, e.g., humans, to be exploited in real-time applications. The proposed method generates multiple textured meshes from multiple RGB-Depth streams, applies a coarse-to-fine registration algorithm and finally merges the separate meshes into a single 3-D surface. Although the Kinect sensor has attracted the attention of many researchers and home enthusiasts and has already appeared in many applications over the Internet, none of the already presented works can produce full 3-D models of moving objects from multiple Kinect streams in real-time. We present the capturing setup, the methodology for its calibration and the details of the proposed algorithm for real-time fusion of multiple meshes. The presented experimental results verify the effectiveness of the approach with respect to the 3-D reconstruction quality, as well as the achieved frame rates.
---
paper_title: A pipeline for building 3D models using depth cameras

paper_content:

In this paper we describe a system for building geometrically consistent 3D models using structured-light depth cameras. While the commercial availability of such devices, i.e. Kinect, has made obtaining depth images easy, the data tends to be corrupted with high levels of noise. In order to work with such noise levels, our approach decouples the problem of scan alignment from that of merging the aligned scans. The alignment problem is solved by using two methods tailored to handle the effects of depth image noise and erroneous alignment estimation. The noisy depth images are smoothed by means of an adaptive bilateral filter that explicitly accounts for the sensitivity of the depth estimation by the scanner. Our robust method overcomes failures due to individual pairwise ICP errors and gives alignments that are accurate and consistent. Finally, the aligned scans are merged using a standard procedure based on the signed distance function representation to build a full 3D model of the object of interest. We demonstrate the performance of our system by building complete 3D models of objects of different physical sizes, ranging from cast-metal busts to a complete model of a small room as well as that of a complex scale model of an aircraft.
---
paper_title: Extracting 3D Mesh Skeletons Using Antipodal Points Locations

paper_content:

Finding the skeleton of a 3D mesh is an essential task for many applications such as mesh animation, tracking, and 3D registeration. In recent years, new technologies in computer vision such as Microsoft Kinect have proven that a mesh skeleton can be useful such as in the case of human machine interactions. To calculate the 3D mesh skeleton, the mesh properties such as topology and its components relations are utilized. In this paper, we propose the usage of a novel algorithm that can efficiently calculate a vertex antipodal point. A vertex antipodal point is the diametrically opposite point that belongs to the same mesh. The set of centers of the connecting lines between each vertex and its antipodal point represents the 3D mesh desired skeleton. Post processing is completed for smoothing and fitting centers into optimized skeleton parts. The algorithm is tested on different classes of 3D objects and produced efficient results that are comparable with the literature. The algorithm has the advantages of producing high quality skeletons as it preserves details. This is suitable for applications where the mesh skeleton mapping is required to be kept as much as possible.
---
paper_title: 3D image reconstruction and human body tracking using stereo vision and Kinect technology

paper_content:

Kinect is a recent technology used for motion detection and human body tracking designed for a video game console. In this study, we explore two different types of 3D image reconstruction methods to achieve a new method for faster and higher quality 3D images. Generating depth perception information using high quality stereo image textures is computationally heavy and inefficient. On the other hand, depth information can be obtained very fast using Kinect but the overall 3D image quality is not refined and it is low resolution. Thus, in this study we explore the combination of higher quality images on a webcam and faster computation of depth information on Kinect in order to create an efficient and enhanced 3D image reconstruction system. This high resolution system has a broad range of applications including 3D motion sensing of human body, hands tracking and finger gestures.
---
paper_title: Using Dynamic Time Warping to Find Patterns in Time Series

paper_content:

Knowledge discovery in databases presents many interesting challenges within the context of providing computer tools for exploring large data archives. Electronic data repositories are growing quickly and contain data from commercial, scientific, and other domains. Much of this data is inherently temporal, such as stock prices or NASA telemetry data. Detecting patterns in such data streams or time series is an important knowledge discovery task. This paper describes some preliminary experiments with a dynamic programming approach to the problem. The pattern detection algorithm is based on the dynamic time warping technique used in the speech recognition field.
---
paper_title: Knee rehabilitation using an intelligent robotic system

paper_content:

There is an increasing trend in using robots for medical purposes. One specific area is the rehabilitation. There are some commercial exercise machines used for rehabilitation purposes. However, these machines have limited use because of their insufficient motion freedom. In addition, these types of machines are not actively controlled and therefore can not accommodate complicated exercises required during rehabilitation. In this study, a rule based intelligent control methodology is proposed to imitate the faculties of an experienced physiotherapist. These involve interpretation of patient reactions, storing the information received, acting according to the available data, and learning from the previous experiences. Robot manipulator is driven by a servo motor and controlled by a computer using force/torque and position sensor information. Impedance control technique is selected for the force control.
---
paper_title: Rule-based approach to recognizing human body poses and gestures in real time

paper_content:

In this paper we propose a classifier capable of recognizing human body static poses and body gestures in real time. The method is called the gesture description language (GDL). The proposed methodology is intuitive, easily thought and reusable for any kind of body gestures. The very heart of our approach is an automated reasoning module. It performs forward chaining reasoning (like a classic expert system) with its inference engine every time new portion of data arrives from the feature extraction library. All rules of the knowledge base are organized in GDL scripts having the form of text files that are parsed with a LALR-1 grammar. The main novelty of this paper is a complete description of our GDL script language, its validation on a large dataset (1,600 recorded movement sequences) and the presentation of its possible application. The recognition rate for examined gestures is within the range of 80.5---98.5 %. We have also implemented an application that uses our method: it is a three-dimensional desktop for visualizing 3D medical datasets that is controlled by gestures recognized by the GDL module.
---
paper_title: SomaTech: an exploratory interface for altering movement habits

paper_content:

We propose SomaTech, a Kinect-based system that encourages users to expand understanding and awareness of their everyday movements. The system creates real-time auditory feedback based on the user's whole action, aiming toward re-education of habitual, potentially unsound movement patterns which are often ingrained within the brain. To do this, we draw inspiration from the field of somatics, which has well-studied prophylactic benefits. Our initial evaluation shows promising results that users become more aware of movement choices and are able to improve their efficiency after using the system.
---
paper_title: Fuzzy Rule-Based Hand Gesture Recognition

paper_content:

This paper introduces a fuzzy rule-based method for the recognition of hand gestures acquired from a data glove, with an application to the recognition of some sample hand gestures of LIBRAS, the Brazilian Sign Language. The method uses the set of angles of finger joints for the classification of hand configurations, and classifications of segments of hand gestures for recognizing gestures. The segmentation of gestures is based on the concept of monotonic gesture segment, sequences of hand configurations in which the variations of the angles of the finger joints have the same sign (non-increasing or non-decreasing). Each gesture is characterized by its list of monotonic segments. The set of all lists of segments of a given set of gestures determine a set of finite automata, which are able to recognize every such gesture.
---
paper_title: Rule based realtime motion assessment for rehabilitation exercises

paper_content:

In this paper, we describe a rule based approach to realtime motion assessment of rehabilitation exercises. We use three types of rules to define each exercise: (1) dynamic rules, with each rule specifying a sequence of monotonic segments of the moving joint or body segment, (2) static rules for stationary joints or body segments, and (3) invariance rules that dictate the requirements of moving joints or body segments. A finite state machine based approach is used in dynamic rule specification and realtime assessment. In addition to the typical advantages of the rule based approach, such as realtime motion assessment with specific feedback, our approach has the following advantages: (1) increased reusability of the defined rules as well as the rule assessment engine facilitated by a set of generic rule elements; (2) increased customizability of the rules for each exercise enabled by the use of a set of generic rule elements and the use of extensible rule encoding method; and (3) increased robustness without relying on expensive statistical algorithms to tolerate motion sensing errors and subtle patient errors.
---
paper_title: KinWrite: Handwriting-Based Authentication Using Kinect

paper_content:

Password-based authentication is easy to use but its security is bounded by how much a user can remember. Biometrics-based authentication requires no memorization but ‘resetting’ a biometric password may not always be possible. In this paper, we propose a user-friendly authentication system (KinWrite) that allows users to choose arbitrary, short and easy-to-memorize passwords while providing resilience to password cracking and password theft. KinWrite lets users write their passwords in 3D space and captures the handwriting motion using a low cost motion input sensing device—Kinect. The low resolution and noisy data captured by Kinect, combined with low consistency of in-space handwriting, have made it challenging to verify users. To overcome these challenges, we exploit the Dynamic Time Warping (DTW) algorithm to quantify similarities between handwritten passwords. Our experimental results involving 35 signatures from 18 subjects and a brute-force attacker have shown that KinWrite can achieve a 100% precision and a 70% recall (the worst case) for verifying honest users, encouraging us to carry out a much larger scale study towards designing a foolproof system.
---
paper_title: One shot learning gesture recognition from RGBD images

paper_content:

We present a system to classify the gesture from only one learning example. The inputs are duo-modality, i.e. RGB and depth sensor from Kinect. Our system performs morphological denoising on depth images and automatically segments the temporal boundaries. Features are extracted based on Extended-Motion-History-Image (Extended-MHI) and the Multi-view Spectral Embedding (MSE) algorithm is used to fuse duo modalities in a physically meaningful manner. Our approach achieves less than 0.3 in Levenshtein distance in CHALEARN Gesture Challenge validation batches [1].
---
paper_title: A motion classifier for Microsoft Kinect

paper_content:

In this paper we proposed a motion classifier using Microsoft Kinect. The main algorithm was based on Dynamic Time Warping (DTW). We tested the classifier with 7 motions. The experimental results showed that DTW coupled with human joint positions captured by Microsoft Kinect were capable of classifying the selected motions.
---
paper_title: Personal Rehabilitation Exercise Assistant with Kinect and Dynamic Time Warping

paper_content:

presence of a professional may cause adverse event or lead to secondary injury. In this paper, we describe our development of a Kinect-based system - KHRD using Dynamic Time Warping (DTW) algorithm and fuzzy logic for ensuring home-based rehabilitation. The KHRD allows a patient perform a prescribed exercise with the presence of a professional. The exercise performed will then be recorded as a base for evaluating the patient's rehabilitation exercise at home. The outcomes of the evaluation can be used as a reference for the patient to validate his/her exercise and to prevent adverse events. A summary report of the outcomes may also be uploaded to a cloud setting for physicians to monitor the patient's progress and adjust the prescription. B. Kinect
---
paper_title: Depth camera based hand gesture recognition and its applications in Human-Computer-Interaction

paper_content:

Of various Human-Computer-Interactions (HCI), hand gesture based HCI might be the most natural and intuitive way to communicate between people and machines, since it closely mimics how human interact with each other. Its intuitiveness and naturalness have spawned many applications in exploring large and complex data, computer games, virtual reality, health care, etc. Although the market for hand gesture based HCI is huge, building a robust hand gesture recognition system remains a challenging problem for traditional vision-based approaches, which are greatly limited by the quality of the input from optical sensors. [16] proposed a novel dissimilarity distance metric for hand gesture recognition using Kinect sensor, called Finger-Earth Mover's Distance (FEMD). In this paper, we compare the performance in terms of speed and accuracy between FEMD and traditional corresponding-based shape matching algorithm, Shape Context. And then we introduce several HCI applications built on top of a accurate and robust hand gesture recognition system based on FEMD. This hand gesture recognition system performs robustly despite variations in hand orientation, scale or articulation. Moreover, it works well in uncontrolled environments with background clusters. We demonstrate that this robust hand gesture recognition system can be a key enabler for numerous hand gesture based HCI systems.
---
paper_title: MotionMA: motion modelling and analysis by demonstration

paper_content:

Particularly in sports or physical rehabilitation, users have to perform body movements in a specific manner for the exercises to be most effective. It remains a challenge for experts to specify how to perform such movements so that an automated system can analyse further performances of it. In a user study with 10 participants we show that experts' explicit estimates do not correspond to their performances. To address this issue we present MotionMA, a system that: (1) automatically extracts a model of movements demonstrated by one user, e.g. a trainer, (2) assesses the performance of other users repeating this movement in real time, and (3) provides real-time feedback on how to improve their performance. We evaluated the system in a second study in which 10 other participants used the system to demonstrate arbitrary movements. Our results demonstrate that MotionMA is able to extract an accurate movement model to spot mistakes and variations in movement execution.
---
paper_title: A logical calculus of the ideas immanent in nervous activity

paper_content:

Abstract Because of the “all-or-none” character of nervous activity, neural events and the relations among them can be treated by means of propositional logic. It is found that the behavior of every net can be described in these terms, with the addition of more complicated logical means for nets containing circles; and that for any logical expression satisfying certain conditions, one can find a net behaving in the fashion it describes. It is shown that many particular choices among possible neurophysiological assumptions are equivalent, in the sense that for every net behaving under one assumption, there exists another net which behaves under the other and gives the same results, although perhaps not in the same time. Various applications of the calculus are discussed.
---
paper_title: Support-vector networks

paper_content:

Thesupport-vector network is a new learning machine for two-group classification problems. The machine conceptually implements the following idea: input vectors are non-linearly mapped to a very high-dimension feature space. In this feature space a linear decision surface is constructed. Special properties of the decision surface ensures high generalization ability of the learning machine. The idea behind the support-vector network was previously implemented for the restricted case where the training data can be separated without errors. We here extend this result to non-separable training data.High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated. We also compare the performance of the support-vector network to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
---
paper_title: A Kinect-based golf swing classification system using HMM and Neuro-Fuzzy

paper_content:

This study displays a method of scoring time-sequential postures of golf swing. Correct posture of golf swing is the most important skill for golfer training. In this paper, firstly, a game controller, Kinect, is used to capture the 3D skeleton coordination of a golfer while swing is performed. Secondly, the time-sequential posture of golf swing features has been extracted. Thirdly, a HMM-NF model is used for scoring, which combines ability of HMM model for temporal data modeling with that of Fuzzy Neural Network for fuzz rule modeling and fuzzy defined in a fuzzy (I am not sure on this!!!). Results have shown that the proposed methods can be implemented to identify and score the golf swing effectively with up to 80% accuracy rate.
---
paper_title: Real-time dynamic gesture recognition system based on depth perception for robot navigation

paper_content:

Natural human robot interaction based on the dynamic hand gesture is becoming a popular research topic in the past few years. The traditional dynamic gesture recognition methods are usually restricted by the factors of illumination condition, varying color and cluttered background. The recognition performance can be improved by using the hand-wearing devices but this is not a natural and barrier-free interaction. To overcome these shortcomings, the depth perception algorithm based on the Kinect depth sensor is introduced to carry out 3D hand tracking. We propose a novel start/end point detection method for segmenting the 3D hand gesture from the hand motion trajectory. Then Hidden Markov Models (HMMs) are implemented to model and classify the hand gesture sequences and the recognized gestures are converted to control commands for the interaction with the robot. Seven different hand gestures performed by two hands can sufficiently navigate the robot. Experiments show that the proposed dynamic hand gesture interaction system can work effectively in the complex environment and in real-time with an average recognition rate of 98.4%. And further experiments for the robot navigation also verify the robustness of our system.
---
paper_title: Unstructured Human Activity Detection from RGBD Images

paper_content:

Being able to detect and recognize human activities is essential for several applications, including personal assistive robotics. In this paper, we perform detection and recognition of unstructured human activity in unstructured environments. We use a RGBD sensor (Microsoft Kinect) as the input sensor, and compute a set of features based on human pose and motion, as well as based on image and point-cloud information. Our algorithm is based on a hierarchical maximum entropy Markov model (MEMM), which considers a person's activity as composed of a set of sub-activities. We infer the two-layered graph structure using a dynamic programming approach. We test our algorithm on detecting and recognizing twelve different activities performed by four people in different environments, such as a kitchen, a living room, an office, etc., and achieve good performance even when the person was not seen before in the training set.1
---
paper_title: A real-time system to recognize static gestures of Brazilian sign language (libras) alphabet using Kinect

paper_content:

In this paper we present a system - called Gesture User Interface (GestureUI) - to recognize static gestures of the Brazilian Sign Language (Libras) in real-time out of a continuous video stream using the depth information captured by a Kinect controller. We focus on handling small sets of gestures (A, E, I, O, U) and (B, C, F, L, V), processing them in two steps: Segmentation and Classification. For the Segmentation we propose a Virtual Wall and Libras-specific heuristics to improve the hand tracking. For the classification we use a Multi-Layer Perceptron trained by the system and present an arm cutting algorithm that improved the recognition rate from 67.4% and 75.4% to 100% for both gesture sets. Finally, we evaluated the processing performance of the overall system and proof that it is able to process frames at 62.5Hz with an Intel i7 processor, which is more than twice as fast as the Kinect frame capturing rate.
---
paper_title: A logical calculus of the ideas immanent in nervous activity

paper_content:

Abstract Because of the “all-or-none” character of nervous activity, neural events and the relations among them can be treated by means of propositional logic. It is found that the behavior of every net can be described in these terms, with the addition of more complicated logical means for nets containing circles; and that for any logical expression satisfying certain conditions, one can find a net behaving in the fashion it describes. It is shown that many particular choices among possible neurophysiological assumptions are equivalent, in the sense that for every net behaving under one assumption, there exists another net which behaves under the other and gives the same results, although perhaps not in the same time. Various applications of the calculus are discussed.
---
paper_title: Learning internal representations by error propagation

paper_content:

This chapter contains sections titled: The Problem, The Generalized Delta Rule, Simulation Results, Some Further Generalizations, Conclusion
---
paper_title: Real-time hand gesture recognition using complex-valued neural network (CVNN)

paper_content:

Computer vision system is one of the newest approaches for human computer interaction. Recently, the direct use of our hands as natural input devices has shown promising progress. Toward this progress, we introduce a hand gesture recognition system in this study to recognize real time gesture in unconstrained environments. The system consists of three components: real time hand tracking, hand-tree construction, and hand gesture recognition. Our main contribution includes: (1) a simple way to represent the hand gesture after applying thinning algorithm to the image, and (2) using a model of complex-valued neural network (CVNN) for real-valued classification. We have tested our system to 26 different gestures to evaluate the effectiveness of our approach. The results show that the classification ability of single-layered CVNN on unseen data is comparable to the conventional real-valued neural network (RVNN) having one hidden layer. Moreover, convergence of the CVNN is much faster than that of the RVNN in most cases.
---
paper_title: Monitoring of the Human Motor Functions Rehabilitation by Neural Networks Based System with Kinect Sensor

paper_content:

Abstract Application of the Kinect sensor for the automatic monitoring and supervision of the human motor functions rehabilitation is considered in this paper. Information about position of patient limbs provided by Kinect sensor is analyzed by the neural networks based system which determines if therapeutic exercise is performed in correct or incorrect way.
---
paper_title: Gesture unit segmentation using support vector machines: segmenting gestures from rest positions

paper_content:

Gesture analysis has been widely used for developing new methods of human-computer interaction. The advancement reached in the gesture analysis area is also motivating its application to automate tasks related to discourse analysis, such as the gesture phases segmentation task. In this paper, we present an initiative that aims at segmenting gestures, especially considering the "units" -- the larger grain involved in gesture phases segmentation. Thereunto, we have captured the gestures using a Xbox Kinect™ device, modeled the problem as a classification task, and applied Support Vector Machines. Moreover, aiming at taking advantage from the temporal aspects involved in the problem, we have used several types of data pre-processing in order to consider time domain and frequency domain features.
---
paper_title: The Nature of Statistical Learning Theory

paper_content:

Setting of the learning problem consistency of learning processes bounds on the rate of convergence of learning processes controlling the generalization ability of learning processes constructing learning algorithms what is important in learning theory?.
---
paper_title: Human gesture recognition using Kinect camera

paper_content:

In this paper, we propose a comparison of human gesture recognition using data mining classification methods in video streaming. In particular, we are interested in a specific stream of vector of twenty body-joint positions which are representative of the human body captured by Kinect camera. The recognized gesture patterns of the study are stand, sit down, and lie down. Classification methods chosen for comparison study are backpropagation neural network, support vector machine, decision tree, and naive Bayes. Experimental results have shown that the backpropagation neural network method outperforms other classification methods and can achieve recognition with 100% accuracy. Moreover, the average accuracy of all classification methods used in this study is 93.72%, which confirms the high potential of using the Kinect camera in human body recognition applications. Our future work will use the knowledge obtained from these classifiers in time series analysis of gesture sequence for detecting fall motion in a smart home system.
---
paper_title: Support-vector networks

paper_content:

Thesupport-vector network is a new learning machine for two-group classification problems. The machine conceptually implements the following idea: input vectors are non-linearly mapped to a very high-dimension feature space. In this feature space a linear decision surface is constructed. Special properties of the decision surface ensures high generalization ability of the learning machine. The idea behind the support-vector network was previously implemented for the restricted case where the training data can be separated without errors. We here extend this result to non-separable training data.High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated. We also compare the performance of the support-vector network to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
---
paper_title: Real-Time Gesture Recognition from Depth Data through Key Poses Learning and Decision Forests

paper_content:

Human gesture recognition is a challenging task with many applications. The popularization of real time depth sensors even diversifies potential applications to end-user natural user interface (NUI). The quality of such NUI highly depends on the robustness and execution speed of the gesture recognition. This work introduces a method for real-time gesture recognition from a noisy skeleton stream, such as the ones extracted from Kinect depth sensors. Each pose is described using a tailored angular representation of the skeleton joints. Those descriptors serve to identify key poses through a multi-class classifier derived from Support Vector learning machines. The gesture is labeled on-the-fly from the key pose sequence through a decision forest, that naturally performs the gesture time warping and avoids the requirement for an initial or neutral pose. The proposed method runs in real time and shows robustness in several experiments.
---
paper_title: An introduction to recursive partitioning: Rationale, application, and characteristics of classification and regression trees, bagging, and random forests.

paper_content:

Recursive partitioning methods have become popular and widely used tools for nonparametric regression and classification in many scientific fields. Especially random forests, which can deal with large numbers of predictor variables even in the presence of complex interactions, have been applied successfully in genetics, clinical medicine, and bioinformatics within the past few years. High-dimensional problems are common not only in genetics, but also in some areas of psychological research, where only a few subjects can be measured because of time or cost constraints, yet a large amount of data is generated for each subject. Random forests have been shown to achieve a high prediction accuracy in such applications and to provide descriptive variable importance measures reflecting the impact of each variable in both main effects and interactions. The aim of this work is to introduce the principles of the standard recursive partitioning methods as well as recent methodological improvements, to illustrate their usage for low and high-dimensional data exploration, but also to point out limitations of the methods and potential pitfalls in their practical application. Application of the methods is illustrated with freely available implementations in the R system for statistical computing.
---
paper_title: Real-Time Gesture Recognition from Depth Data through Key Poses Learning and Decision Forests

paper_content:

Human gesture recognition is a challenging task with many applications. The popularization of real time depth sensors even diversifies potential applications to end-user natural user interface (NUI). The quality of such NUI highly depends on the robustness and execution speed of the gesture recognition. This work introduces a method for real-time gesture recognition from a noisy skeleton stream, such as the ones extracted from Kinect depth sensors. Each pose is described using a tailored angular representation of the skeleton joints. Those descriptors serve to identify key poses through a multi-class classifier derived from Support Vector learning machines. The gesture is labeled on-the-fly from the key pose sequence through a decision forest, that naturally performs the gesture time warping and avoids the requirement for an initial or neutral pose. The proposed method runs in real time and shows robustness in several experiments.
---
paper_title: Simultaneous categorical and spatio-temporal 3D gestures using Kinect

paper_content:

Recent technological advances have led to an increasing popularity of 3D gesture-based interfaces, in particular in gaming and entertainment consoles. However, unlike 2D gestures, which have been successfully utilized in many multi-touch devices, developing a 3D gesture-based interface is not an easy endeavor. Reasons include the complexity of capturing human movements in 3D and the difficulties associated with recognizing gestures from human motion data. In this work, we target the latter problem by proposing a novel gesture recognition technique for skeletal input data that simultaneously allows for categorical and spatio-temporal gestures. In other words, it recognizes the gesture type and the relative pose within a gesture at the same time. Moreover, our method can learn gestures that are most appropriate for the user from examples. In order to avoid the need for user-specific training, we further propose and evaluate several types of feature representations for human pose data. We argue how our approach can facilitate the development of a customizable 3D gesture-based interface and explore possibilities in order to smoothly integrate the proposed recognition approach into available component-based user interface frameworks. Besides a quantitative evaluation, we present a user study in the scenario of a 3D gesture-based interface for an intra-operative medical image viewer. Our studies support the applicability of our method for developing 3D gesture-based interfaces in practice.
---
paper_title: A least squares regression framework on manifolds and its application to gesture recognition

paper_content:

Least squares regression is a basic approach for statistical analysis. However, its simplicity has often led to researchers overlooking it for complex recognition problems. In this paper, we present a nonlinear regression framework on manifolds for gesture recognition. Our method is developed based upon two key attributes: underlying geometry and least squares fitting. The former attribute is vital since geometry characterizes the space for classification while the latter exhibits a simple estimation model. Considering geometric properties, we formulate least squares regression as a composite function. This gives a natural extension from Euclidean space to manifolds. Our experiments show that the proposed framework achieves state-of-the-art results on the standard hand gesture and body gesture datasets. Our method also generalizes well on the one-shot-learning CHALEARN gesture challenge.
---
paper_title: Principal Component Analysis

paper_content:

When large multivariate datasets are analyzed, it is often desirable to reduce their dimensionality. Principal component analysis is one technique for doing this. It replaces the p original variables by a smaller number, q, of derived variables, the principal components, which are linear combinations of the original variables. Often, it is possible to retain most of the variability in the original variables with q very much smaller than p. Despite its apparent simplicity, principal component analysis has a number of subtleties, and it has many uses and extensions. A number of choices associated with the technique are briefly discussed, namely, covariance or correlation, how many components, and different normalization constraints, as well as confusion with factor analysis. Various uses and extensions are outlined. ::: ::: ::: Keywords: ::: ::: dimension reduction; ::: factor analysis; ::: multivariate analysis; ::: variance maximization
---
paper_title: Human action recognition using Action Trait Code

paper_content:

Recognizing actions having similar movements is a challenging problem. Human action understanding task is divided into two issues in this paper. One is a classical action recognition task where we employ a probabilistic model to learn and recognize human actions. The other is action categorization task where we classify actions based on quantized human movement. An approach called Action Trait Code (ATC) for human action classification is proposed to represent an action with a set of velocity types derived by the averages velocity of each body part. An effective graph model based on ATC classification is employed for learning and recognizing human actions. To examine recognition accuracy, we evaluate our approach on Cornell Kinect Activity Database and compare with a hierarchical maximum entropy Markov model (MEMM). Besides, the results on self-collected action database demonstrate that the proposed approach not only successfully achieves high recognition accuracy but also performs in real-time.
---
paper_title: Extraction of Discriminative Patterns from Skeleton Sequences for Human Action Recognition

paper_content:

Emergence of novel techniques, such as the invention of MS Kinect, enables reliable extraction of human skeletons from action videos. Taking skeleton data as inputs, we propose an approach in this paper to extract the discriminative patterns for efficient human action recognition. Each action is considered to consist of a series of unit actions, each of which is represented by a pattern. Given a skeleton sequence, we first automatically extract the key-frames for unit actions, and then label them as different patterns. We further use a statistical metric to evaluate the discrim- inative capability of each pattern, and define the bag of reliable patterns as local features for action recognition. Experimental results show that the extracted local descriptors could provide very high accuracy in the action recognition, which demonstrate the efficiency of our method in extracting discriminative patterns.
---
paper_title: People tracking in RGB-D data with on-line boosted target models

paper_content:

People tracking is a key component for robots that are deployed in populated environments. Previous works have used cameras and 2D and 3D range finders for this task. In this paper, we present a 3D people detection and tracking approach using RGB-D data. We combine a novel multi-cue person detector for RGB-D data with an on-line detector that learns individual target models. The two detectors are integrated into a decisional framework with a multi-hypothesis tracker that controls on-line learning through a track interpretation feedback. For on-line learning, we take a boosting approach using three types of RGB-D features and a confidence maximization search in 3D space. The approach is general in that it neither relies on background learning nor a ground plane assumption. For the evaluation, we collect data in a populated indoor environment using a setup of three Microsoft Kinect sensors with a joint field of view. The results demonstrate reliable 3D tracking of people in RGB-D data and show how the framework is able to avoid drift of the on-line detector and increase the overall tracking performance.
---
paper_title: Learning human activities and object affordances from RGB-D videos

paper_content:

Understanding human activities and object affordances are two very important skills, especially for personal robots which operate in human environments. In this work, we consider the problem of extracting a descriptive labeling of the sequence of sub-activities being performed by a human, and more importantly, of their interactions with the objects in the form of associated affordances. Given a RGB-D video, we jointly model the human activities and object affordances as a Markov random field where the nodes represent objects and sub-activities, and the edges represent the relationships between object affordances, their relations with sub-activities, and their evolution over time. We formulate the learning problem using a structural support vector machine (SSVM) approach, where labelings over various alternate temporal segmentations are considered as latent variables. We tested our method on a challenging dataset comprising 120 activity videos collected from 4 subjects, and obtained an accuracy of 79.4% for affordance, 63.4% for sub-activity and 75.0% for high-level activity labeling. We then demonstrate the use of such descriptive labeling in performing assistive tasks by a PR2 robot.
---
paper_title: A category-level 3-D object dataset: Putting the Kinect to work

paper_content:

Recent proliferation of a cheap but quality depth sensor, the Microsoft Kinect, has brought the need for a challenging category-level 3D object detection dataset to the fore. We review current 3D datasets and find them lacking in variation of scenes, categories, instances, and viewpoints. Here we present our dataset of color and depth image pairs, gathered in real domestic and office environments. It currently includes over 50 classes, with more images added continuously by a crowd-sourced collection effort. We establish baseline performance in a PASCAL VOC-style detection task, and suggest two ways that inferred world size of the object may be used to improve detection. The dataset and annotations can be downloaded at http://www.kinectdata.com.
---
paper_title: People detection in RGB-D data

paper_content:

People detection is a key issue for robots and intelligent systems sharing a space with people. Previous works have used cameras and 2D or 3D range finders for this task. In this paper, we present a novel people detection approach for RGB-D data. We take inspiration from the Histogram of Oriented Gradients (HOG) detector to design a robust method to detect people in dense depth data, called Histogram of Oriented Depths (HOD). HOD locally encodes the direction of depth changes and relies on an depth-informed scale-space search that leads to a 3-fold acceleration of the detection process. We then propose Combo-HOD, a RGB-D detector that probabilistically combines HOD and HOG. The experiments include a comprehensive comparison with several alternative detection approaches including visual HOG, several variants of HOD, a geometric person detector for 3D point clouds, and an Haar-based AdaBoost detector. With an equal error rate of 85% in a range up to 8m, the results demonstrate the robustness of HOD and Combo-HOD on a real-world data set collected with a Kinect sensor in a populated indoor environment.
---
paper_title: Unstructured Human Activity Detection from RGBD Images

paper_content:

Being able to detect and recognize human activities is essential for several applications, including personal assistive robotics. In this paper, we perform detection and recognition of unstructured human activity in unstructured environments. We use a RGBD sensor (Microsoft Kinect) as the input sensor, and compute a set of features based on human pose and motion, as well as based on image and point-cloud information. Our algorithm is based on a hierarchical maximum entropy Markov model (MEMM), which considers a person's activity as composed of a set of sub-activities. We infer the two-layered graph structure using a dynamic programming approach. We test our algorithm on detecting and recognizing twelve different activities performed by four people in different environments, such as a kitchen, a living room, an office, etc., and achieve good performance even when the person was not seen before in the training set.1
---
paper_title: A dataset of Kinect-based 3D scans

paper_content:

Hereby, a new publicly available 3D reconstruction-oriented dataset is presented. It consists of multi-view range scans of small-sized objects using a turntable. Range scans were captured using a Microsoft Kinect sensor, as well as an accurate laser scanner (Vivid VI-700 Non-contact 3D Digitizer), whose reconstructions can serve as ground-truth data. The construction of this dataset was motivated by the lack of a relevant Kinect dataset, despite the fact that Kinect has attracted the attention of many researchers and home enthusiasts. Thus, the core idea behind the construction of this dataset, is to allow the validation of 3D surface reconstruction methodologies for point sets extracted using Kinect sensors. The dataset consists of multi-view range scans of 59 objects, along with the necessary calibration information that can be used for experimentation in the field of 3D reconstruction from Kinect depth data. Two well-known 3D reconstruction methods were selected and applied on the dataset, in order to demonstrate its applicability in the 3D reconstruction field, as well as the challenges that arise. Additionally, the appropriate 3D reconstruction evaluation methodology is presented. Finally, as the dataset comes in classes of similar objects, it can also be used for classification purposes, using the provided 2.5D/3D features.
---

Title: A Survey of Applications and Human Motion Recognition with Microsoft Kinect

Section 1: Introduction
Description 1: Write about the introduction of the Microsoft Kinect technology, its popularity, and how it has been extended and studied in various fields and computer vision-based human motion tracking and recognition.

Section 2: The Kinect Technology
Description 2: Explain the main features of the Kinect sensors, the underlying depth sensing technologies, and the two versions of the Kinect sensor (Kinect v1 and Kinect v2).

Section 3: Applications of the Kinect Technology
Description 3: Review various applications of the Microsoft Kinect sensor, including gaming, healthcare, education, robotics, sign language recognition, and retail services.

Section 4: Healthcare Applications
Description 4: Detail the applications of Kinect in healthcare, with a focus on physical rehabilitation exercises, medical operating room assistance, and fall detection and prevention.

Section 5: Virtual Reality and Gaming
Description 5: Discuss how Kinect is used in virtual reality and gaming, enabling real-time user interaction with augmented objects.

Section 6: Natural User Interface
Description 6: Explain the development of natural user interfaces that use Kinect for human-computer interaction through gestures and voice commands.

Section 7: Education and Performing Arts
Description 7: Review how Kinect is used in educational contexts and performing arts to engage students and enhance teaching and performing experiences.

Section 8: Robotics Control and Interaction
Description 8: Provide an overview of how Kinect is utilized in navigating, controlling, and interacting with robotics.

Section 9: Speech and Sign Language Recognition
Description 9: Survey the use of Kinect in speech and sign language recognition applications.

Section 10: Retail Services
Description 10: Explain ways in which Kinect technology is beneficial to retail services, such as analyzing customer behavior and providing virtual try-on experience.

Section 11: Workplace Safety Training
Description 11: Describe Kinect-based systems for workplace safety training, including injury prevention and ergonomics assessment.

Section 12: 3D Reconstruction
Description 12: Discuss the challenges and solutions in using Kinect for 3D reconstruction, including noise handling and alignment of scan data.

Section 13: Human Motion Recognition with Microsoft Kinect
Description 13: Provide a detailed review of the models and algorithms used for recognizing human gestures and activities using Kinect.

Section 14: Publicly Available Kinect Datasets
Description 14: Compile a list and brief description of publicly available Kinect datasets for research in motion analysis and computer vision.

Section 15: Conclusion
Description 15: Summarize the comprehensive coverage of Kinect applications, motion recognition techniques, and the importance of feature selection and dataset availability.