AutoDepthNet: High Frame Rate Depth Map Reconstruction using Commodity
  Depth and RGB Cameras
Peyman Gholami Robert Xiao
Depth cameras have found applications in diverse fields, such as computer
vision, artificial intelligence, and video gaming. However, the high latency
and low frame rate of existing commodity depth cameras impose limitations on
their applications. We propose a fast and accurate depth map reconstruction
technique to reduce latency and increase the frame rate in depth cameras. Our
approach uses only a commodity depth camera and color camera in a hybrid camera
setup; our prototype is implemented using a Kinect Azure depth camera at 30 fps
and a high-speed RGB iPhone 11 Pro camera captured at 240 fps. The proposed
network, AutoDepthNet, is an encoder-decoder model that captures frames from
the high-speed RGB camera and combines them with previous depth frames to
reconstruct a stream of high frame rate depth maps. On GPU, with a 480 x 270
output resolution, our system achieves an inference time of 8 ms, enabling
real-time use at up to 200 fps with parallel processing. AutoDepthNet can
estimate depth values with an average RMS error of 0.076, a 44.5% improvement
compared to an optical flow-based comparison method. Our method can also
improve depth map quality by estimating depth values for missing and
invalidated pixels. The proposed method can be easily applied to existing depth
cameras and facilitates the use of depth cameras in applications that require
high-speed depth estimation. We also showcase the effectiveness of the
framework in upsampling different sparse datasets e.g. video object
segmentation. As a demonstration of our method, we integrated our framework
into existing body tracking systems and demonstrated the robustness of the
proposed method in such applications.