Taxonomy of 3D sensors A Survey of State-of-the-Art Consumer 3D-Reconstruction Sensors and Their Field of Applications
---
paper_title: Real-time non-rigid reconstruction using an RGB-D camera

paper_content:

We present a combined hardware and software solution for markerless reconstruction of non-rigidly deforming physical objects with arbitrary shape in real-time. Our system uses a single self-contained stereo camera unit built from off-the-shelf components and consumer graphics hardware to generate spatio-temporally coherent 3D models at 30 Hz. A new stereo matching algorithm estimates real-time RGB-D data. We start by scanning a smooth template model of the subject as they move rigidly. This geometric surface prior avoids strong scene assumptions, such as a kinematic human skeleton or a parametric shape model. Next, a novel GPU pipeline performs non-rigid registration of live RGB-D data to the smooth template using an extended non-linear as-rigid-as-possible (ARAP) framework. High-frequency details are fused onto the final mesh using a linear deformation model. The system is an order of magnitude faster than state-of-the-art methods, while matching the quality and robustness of many offline algorithms. We show precise real-time reconstructions of diverse scenes, including: large deformations of users' heads, hands, and upper bodies; fine-scale wrinkles and folds of skin and clothing; and non-rigid interactions performed by users on flexible objects such as toys. We demonstrate how acquired models can be used for many interactive scenarios, including re-texturing, online performance capture and preview, and real-time shape and motion re-targeting.
---
paper_title: Visual odometry and mapping for autonomous flight using an RGB-D camera

paper_content:

RGB-D cameras provide both a color image and per-pixel depth estimates. The richness of their data and the recent development of low-cost sensors have combined to present an attractive opportunity for mobile robotics research. In this paper, we describe a system for visual odometry and mapping using an RGB-D camera, and its application to autonomous flight. By leveraging results from recent state-of-the-art algorithms and hardware, our system enables 3D flight in cluttered environments using only onboard sensor data. All computation and sensing required for local position control are performed onboard the vehicle, reducing the dependence on unreliable wireless links. We evaluate the effectiveness of our system for stabilizing and controlling a quadrotor micro air vehicle, demonstrate its use for constructing detailed 3D maps of an indoor environment, and discuss its limitations.
---
paper_title: 3D shape scanning with a time-of-flight camera

paper_content:

We describe a method for 3D object scanning by aligning depth scans that were taken from around an object with a time-of-flight camera. These ToF cameras can measure depth scans at video rate. Due to comparably simple technology they bear potential for low cost production in big volumes. Our easy-to-use, cost-effective scanning solution based on such a sensor could make 3D scanning technology more accessible to everyday users. The algorithmic challenge we face is that the sensor's level of random noise is substantial and there is a non-trivial systematic bias. In this paper we show the surprising result that 3D scans of reasonable quality can also be obtained with a sensor of such low data quality. Established filtering and scan alignment techniques from the literature fail to achieve this goal. In contrast, our algorithm is based on a new combination of a 3D superresolution method with a probabilistic scan alignment approach that explicitly takes into account the sensor's noise characteristics.
---
paper_title: Monitoring patients in hospital beds using unobtrusive depth sensors

paper_content:

We present an approach for patient activity recognition in hospital rooms using depth data collected using a Kinect sensor. Depth sensors such as the Kinect ensure that activity segmentation is possible during day time as well as night while addressing the privacy concerns of patients. It also provides a technique to remotely monitor patients in a non-intrusive manner. An existing fall detection algorithm is currently generating fall alerts in several rooms in the University of Missouri Hospital (MUH). In this paper we describe a technique to reduce false alerts such as pillows falling off the bed or equipment movement. We do so by detecting the presence of the patient in the bed for the times when the fall alert is generated. We test our algorithm on 96 hours obtained in two hospital rooms from MUH.
---
paper_title: Controller-free exploration of medical image data: Experiencing the Kinect

paper_content:

In this paper, an open-source system for a controller-free, highly interactive exploration of medical images is presented. By using a Microsoft Xbox Kinect™ as the only input device, the system's user interface allows users to interact at a distance through hand and arm gestures. The paper also details the interaction techniques we have designed specifically for the deviceless exploration of medical imaging data. Since the user interface is touch-free and does not require complex calibration steps, it is suitable for use in operating rooms, where non-sterilizable devices cannot be used.
---
paper_title: A Multi-Resolution Approach for an Automated Fusion of Different Low-Cost 3D Sensors

paper_content:

The 3D acquisition of object structures has become a common technique in many fields of work, e.g., industrial quality management, cultural heritage or crime scene documentation. The requirements on the measuring devices are versatile, because spacious scenes have to be imaged with a high level of detail for selected objects. Thus, the used measuring systems are expensive and require an experienced operator. With the rise of low-cost 3D imaging systems, their integration into the digital documentation process is possible. However, common low-cost sensors have the limitation of a trade-off between range and accuracy, providing either a low resolution of single objects or a limited imaging field. Therefore, the use of multiple sensors is desirable. We show the combined use of two low-cost sensors, the Microsoft Kinect and the David laserscanning system, to achieve low-resolved scans of the whole scene and a high level of detail for selected objects, respectively. Afterwards, the high-resolved David objects are automatically assigned to their corresponding Kinect object by the use of surface feature histograms and SVM-classification. The corresponding objects are fitted using an ICP-implementation to produce a multi-resolution map. The applicability is shown for a fictional crime scene and the reconstruction of a ballistic trajectory.
---
paper_title: Study on the use of Microsoft Kinect for robotics applications

paper_content:

The Microsoft X-Box Kinect Sensor is a revolutionary new depth camera that is used in the gaming industry to capture motions of people and players efficiently using the technology of an RGB camera and infrared camera to differentiate depth. In the Microsoft X-Box, Kinect was used to sense 3D perception of human's motions. It can also be used for robotic applications, precisely for indoor navigation through the process of reverse engineering. Certain software packages were made available and are open source from “LibFreenect” for Linux machines, Microsoft's Kinect SDK using the Kinect namespace on Visual Studio 2010 Express (C++, C# or Visual Basic), and Google's released “Robotic Operating System (ROS)”. In order to claim that this sensor is capable of taking on such a task, we must be able to investigate thoroughly all factors that contribute to this and at the same time we must be able to understand its limitations to be applied and integrated properly with certain types of robots for accomplishing our purpose of achieving successful indoor navigation using proper algorithms. In this paper, the results from testing the Kinect sensor on an autonomous ground vehicle was given.
---
paper_title: Development of an omnidirectional mobile robot using a RGB-D sensor for indoor navigation

paper_content:

The demand of service robots has increased in recent decades, especially for indoor robotics. Omnidirectional drive system is suitable to work in crowded and small space indoor environment like home, office, hospital etc. In modern robotics, researchers are keen to equip sensing ability into robots, which encourages the development and application of sensors in robot such as gyroscope and camera. Using a RGB-D sensor is a possible method to provide vision ability to a mobile robot. Moreover, simultaneous localization and mapping (SLAM) is one of popular approach to make robot identify its position by sensing the surrounding environment. In this work, we developed an omnidirectional driving mobile robot using Mecanum wheels, which is equipped a RGB-D camera to obtain the surrounding information. The calibration of the robot including the motors, the velocity and the object depth information were done. Demonstrations of object detection and mapping are included in this paper.
---
paper_title: Seeing Human Weight from a Single RGB-D Image

paper_content:

Human weight estimation is useful in a variety of potential applications, e.g., targeted advertisement, entertainment scenarios and forensic science. However, estimating weight only from color cues is particularly challenging since these cues are quite sensitive to lighting and imaging conditions. In this article, we propose a novel weight estimator based on a single RGB-D image, which utilizes the visual color cues and depth information. Our main contributions are three-fold. First, we construct the W8-RGBD dataset including RGB-D images of different people with ground truth weight. Second, the novel sideview shape feature and the feature fusion model are proposed to facilitate weight estimation. Additionally, we consider gender as another important factor for human weight estimation. Third, we conduct comprehensive experiments using various regression models and feature fusion models on the new weight dataset, and encouraging results are obtained based on the proposed features and models.
---
paper_title: Rgbd mapping: Using depth cameras for dense 3d modeling of indoor environments

paper_content:

RGB-D cameras are novel sensing systems that capture RGB images along with per-pixel depth information. In this paper we investigate how such cameras can be used in the context of robotics, specifically for building dense 3D maps of indoor environments. Such maps have applications in robot navigation, manipulation, semantic mapping, and telepresence. We present RGB-D Mapping, a full 3D mapping system that utilizes a novel joint optimization algorithm combining visual features and shape-based alignment. Visual and depth information are also combined for view-based loop closure detection, followed by pose optimization to achieve globally consistent maps.We evaluate RGB-D Mapping on two large indoor environments, and show that it effectively combines the visual and shape information available from RGB-D cameras.
---
paper_title: Accuracy and Resolution of Kinect Depth Data for Indoor Mapping Applications

paper_content:

Consumer-grade range cameras such as the Kinect sensor have the potential to be used in mapping applications where accuracy requirements are less strict. To realize this potential insight into the geometric quality of the data acquired by the sensor is essential. In this paper we discuss the calibration of the Kinect sensor, and provide an analysis of the accuracy and resolution of its depth data. Based on a mathematical model of depth measurement from disparity a theoretical error analysis is presented, which provides an insight into the factors influencing the accuracy of the data. Experimental results show that the random error of depth measurement increases with increasing distance to the sensor, and ranges from a few millimeters up to about 4 cm at the maximum range of the sensor. The quality of the data is also found to be influenced by the low resolution of the depth measurements.
---
paper_title: KinectFusion: Real-time dense surface mapping and tracking

paper_content:

We present a system for accurate real-time mapping of complex and arbitrary indoor scenes in variable lighting conditions, using only a moving low-cost depth camera and commodity graphics hardware. We fuse all of the depth data streamed from a Kinect sensor into a single global implicit surface model of the observed scene in real-time. The current sensor pose is simultaneously obtained by tracking the live depth frame relative to the global model using a coarse-to-fine iterative closest point (ICP) algorithm, which uses all of the observed depth data available. We demonstrate the advantages of tracking against the growing full surface model compared with frame-to-frame tracking, obtaining tracking and mapping results in constant time within room sized scenes with limited drift and high accuracy. We also show both qualitative and quantitative results relating to various aspects of our tracking and mapping system. Modelling of natural scenes, in real-time with only commodity sensor and GPU hardware, promises an exciting step forward in augmented reality (AR), in particular, it allows dense surfaces to be reconstructed in real-time, with a level of detail and robustness beyond any solution yet presented using passive computer vision.
---
paper_title: Transparent and Specular Object Reconstruction

paper_content:

This state of the art report covers reconstruction methods for transparent and specular objects or phenomena. While the 3D acquisition of opaque surfaces with Lambertian reflectance is a well-studied problem, transparent, refractive, specular and potentially dynamic scenes pose challenging problems for acquisition systems. This report reviews and categorizes the literature in this field. ::: ::: ::: ::: Despite tremendous interest in object digitization, the acquisition of digital models of transparent or specular objects is far from being a solved problem. On the other hand, real-world data is in high demand for applications such as object modelling, preservation of historic artefacts and as input to data-driven modelling techniques. With this report we aim at providing a reference for and an introduction to the field of transparent and specular object reconstruction. ::: ::: ::: ::: We describe acquisition approaches for different classes of objects. Transparent objects/phenomena that do not change the straight ray geometry can be found foremost in natural phenomena. Refraction effects are usually small and can be considered negligible for these objects. Phenomena as diverse as fire, smoke, and interstellar nebulae can be modelled using a straight ray model of image formation. Refractive and specular surfaces on the other hand change the straight rays into usually piecewise linear ray paths, adding additional complexity to the reconstruction problem. Translucent objects exhibit significant sub-surface scattering effects rendering traditional acquisition approaches unstable. Different classes of techniques have been developed to deal with these problems and good reconstruction results can be achieved with current state-of-the-art techniques. However, the approaches are still specialized and targeted at very specific object classes. We classify the existing literature and hope to provide an entry point to this exiting field.
---
paper_title: Voxel Cloud Connectivity Segmentation - Supervoxels for Point Clouds

paper_content:

Unsupervised over-segmentation of an image into regions of perceptually similar pixels, known as super pixels, is a widely used preprocessing step in segmentation algorithms. Super pixel methods reduce the number of regions that must be considered later by more computationally expensive algorithms, with a minimal loss of information. Nevertheless, as some information is inevitably lost, it is vital that super pixels not cross object boundaries, as such errors will propagate through later steps. Existing methods make use of projected color or depth information, but do not consider three dimensional geometric relationships between observed data points which can be used to prevent super pixels from crossing regions of empty space. We propose a novel over-segmentation algorithm which uses voxel relationships to produce over-segmentations which are fully consistent with the spatial geometry of the scene in three dimensional, rather than projective, space. Enforcing the constraint that segmented regions must have spatial connectivity prevents label flow across semantic object boundaries which might otherwise be violated. Additionally, as the algorithm works directly in 3D space, observations from several calibrated RGB+D cameras can be segmented jointly. Experiments on a large data set of human annotated RGB+D images demonstrate a significant reduction in occurrence of clusters crossing object boundaries, while maintaining speeds comparable to state-of-the-art 2D methods.
---
paper_title: RGB-D fusion: Real-time robust tracking and dense mapping with RGB-D data fusion

paper_content:

We present RGB-D Fusion, a framework which robustly tracks and reconstructs dense textured surfaces of scenes and objects by integrating both color and depth images streamed from a RGB-D sensor into a global colored volume in real-time. To handle failure of the ICP-based tracking approach, KinectFusion, due to the lack of sufficient geometric information, we propose a novel approach which registers the input RGB-D image with the colored volume by pho- tometric tracking and geometric alignment. We demonstrate the strengths of the proposed approach compared with the ICP-based approach and show superior performance of our algorithm with real-world data. I. INTRODUCTION Real-time photo-realistic 3D reconstruction of real world scenes and objects plays an important role for a variety of applications such as robotics, augmented reality, human computer interaction, and entertainment. One of the main requirements in the robot interaction with the environment is the 3D reconstruction and representation of real world environments. With the invention of the low-cost Microsoft Kinect sensor as an representative of a consumer-grade range sensing device, the high-resolution range images aligned with pho- tometric information become relatively easily available. The Kinect sensor opens up new possibilities to build a low- cost 3D modeling system with well-studied range mapping technologies.
---
paper_title: Interactive 3D Reconstruction: New Opportunities for Getting CAD-ready Models

paper_content:

A multitude of image-based 3D reconstruction and modeling techniques exist, which have achieved significant success in recent years. However, these techniques still lack certain abilities. For example, current 3D reconstruction techniques cannot decompose an object into its individual subparts. Thus, a printed model will consist of one single monolithic piece, which does not allow composing or decomposing parts, does not allow movable or flexible parts, and does not allow manufacturing the model from multiple different materials like wood, metal, or plastic. I reviewed the work in the research area of 3D reconstruction and provide an analysis of neglected research objectives and current drawbacks. Furthermore, I propose a mock-up of an interactive tool as a guideline for future research which describes a possible architecture, user interfaces, and processing pipeline, to overcome existing drawbacks of 3D reconstruction techniques.
---
paper_title: Rgbd mapping: Using depth cameras for dense 3d modeling of indoor environments

paper_content:

RGB-D cameras are novel sensing systems that capture RGB images along with per-pixel depth information. In this paper we investigate how such cameras can be used in the context of robotics, specifically for building dense 3D maps of indoor environments. Such maps have applications in robot navigation, manipulation, semantic mapping, and telepresence. We present RGB-D Mapping, a full 3D mapping system that utilizes a novel joint optimization algorithm combining visual features and shape-based alignment. Visual and depth information are also combined for view-based loop closure detection, followed by pose optimization to achieve globally consistent maps.We evaluate RGB-D Mapping on two large indoor environments, and show that it effectively combines the visual and shape information available from RGB-D cameras.
---

Title: Taxonomy of 3D sensors: A Survey of State-of-the-Art Consumer 3D-Reconstruction Sensors and Their Field of Applications

Section 1: Introduction
Description 1: Introduce the advent and impact of the Kinect camera, the current state and limitations of available consumer RGB-D cameras, and the motivation for the survey.

Section 2: Comparison of Consumer 3D Sensors
Description 2: Discuss different groups of consumer 3D sensors based on their measurement principles and compare them using attributes like field of view, supported programming languages, price, and price-performance ratio.

Section 3: Structured Light Sensors
Description 3: Provide detailed descriptions and evaluations of various structured light-based 3D sensors, including their specifications, capabilities, and drawbacks.

Section 4: Time of Flight Sensors
Description 4: Offer detailed descriptions and evaluations of various time of flight-based 3D sensors, including their specifications, capabilities, and drawbacks.

Section 5: Application
Description 5: Examine specific research applications of 3D sensors, assessing whether they offer sufficient functionality and quality for tasks such as 3D reconstruction or mapping, supported by test scenarios.

Section 6: Discussion and Outlook
Description 6: Summarize the findings of the survey, discuss existing limitations of consumer 3D sensors for indoor and outdoor applications, and highlight future directions for development and improvement.