Differentiable Registration of Images and LiDAR Point Clouds with
  VoxelPoint-to-Pixel Matching
Junsheng Zhou Baorui Ma Wenyuan Zhang Yi Fang Yu-Shen Liu Zhizhong Han
Cross-modality registration between 2D images from cameras and 3D point
clouds from LiDARs is a crucial task in computer vision and robotic. Previous
methods estimate 2D-3D correspondences by matching point and pixel patterns
learned by neural networks, and use Perspective-n-Points (PnP) to estimate
rigid transformation during post-processing. However, these methods struggle to
map points and pixels to a shared latent space robustly since points and pixels
have very different characteristics with patterns learned in different manners
(MLP and CNN), and they also fail to construct supervision directly on the
transformation since the PnP is non-differentiable, which leads to unstable
registration results. To address these problems, we propose to learn a
structured cross-modality latent space to represent pixel features and 3D
features via a differentiable probabilistic PnP solver. Specifically, we design
a triplet network to learn VoxelPoint-to-Pixel matching, where we represent 3D
elements using both voxels and points to learn the cross-modality latent space
with pixels. We design both the voxel and pixel branch based on CNNs to operate
convolutions on voxels/pixels represented in grids, and integrate an additional
point branch to regain the information lost during voxelization. We train our
framework end-to-end by imposing supervisions directly on the predicted pose
distribution with a probabilistic PnP solver. To explore distinctive patterns
of cross-modality features, we design a novel loss with adaptive-weighted
optimization for cross-modality feature description. The experimental results
on KITTI and nuScenes datasets show significant improvements over the
state-of-the-art methods. The code and models are available at
https://github.com/junshengzhou/VP2P-Match.