3D Implicit Transporter for Temporally Consistent Keypoint Discovery
Chengliang Zhong Yuhang Zheng Yupeng Zheng Hao Zhao Li Yi Xiaodong Mu Ling Wang Pengfei Li Guyue Zhou Chao Yang Xinliang Zhang Jian Zhao
Keypoint-based representation has proven advantageous in various visual and
robotic tasks. However, the existing 2D and 3D methods for detecting keypoints
mainly rely on geometric consistency to achieve spatial alignment, neglecting
temporal consistency. To address this issue, the Transporter method was
introduced for 2D data, which reconstructs the target frame from the source
frame to incorporate both spatial and temporal information. However, the direct
application of the Transporter to 3D point clouds is infeasible due to their
structural differences from 2D images. Thus, we propose the first 3D version of
the Transporter, which leverages hybrid 3D representation, cross attention, and
implicit reconstruction. We apply this new learning system on 3D articulated
objects and nonrigid animals (humans and rodents) and show that learned
keypoints are spatio-temporally consistent. Additionally, we propose a
closed-loop control strategy that utilizes the learned keypoints for 3D object
manipulation and demonstrate its superior performance. Codes are available at
https://github.com/zhongcl-thu/3D-Implicit-Transporter.