EP2P-Loc: End-to-End 3D Point to 2D Pixel Localization for Large-Scale
  Visual Localization
Minjung Kim Junseo Koo Gunhee Kim
Visual localization is the task of estimating a 6-DoF camera pose of a query
image within a provided 3D reference map. Thanks to recent advances in various
3D sensors, 3D point clouds are becoming a more accurate and affordable option
for building the reference map, but research to match the points of 3D point
clouds with pixels in 2D images for visual localization remains challenging.
Existing approaches that jointly learn 2D-3D feature matching suffer from low
inliers due to representational differences between the two modalities, and the
methods that bypass this problem into classification have an issue of poor
refinement. In this work, we propose EP2P-Loc, a novel large-scale visual
localization method that mitigates such appearance discrepancy and enables
end-to-end training for pose estimation. To increase the number of inliers, we
propose a simple algorithm to remove invisible 3D points in the image, and find
all 2D-3D correspondences without keypoint detection. To reduce memory usage
and search complexity, we take a coarse-to-fine approach where we extract
patch-level features from 2D images, then perform 2D patch classification on
each 3D point, and obtain the exact corresponding 2D pixel coordinates through
positional encoding. Finally, for the first time in this task, we employ a
differentiable PnP for end-to-end training. In the experiments on newly curated
large-scale indoor and outdoor benchmarks based on 2D-3D-S and KITTI, we show
that our method achieves the state-of-the-art performance compared to existing
visual localization and image-to-point cloud registration methods.