Elite360D: Towards Efficient 360 Depth Estimation via Semantic- and
  Distance-Aware Bi-Projection Fusion
Hao Ai Lin Wang
360 depth estimation has recently received great attention for 3D
reconstruction owing to its omnidirectional field of view (FoV). Recent
approaches are predominantly focused on cross-projection fusion with
geometry-based re-projection: they fuse 360 images with equirectangular
projection (ERP) and another projection type, e.g., cubemap projection to
estimate depth with the ERP format. However, these methods suffer from 1)
limited local receptive fields, making it hardly possible to capture large FoV
scenes, and 2) prohibitive computational cost, caused by the complex
cross-projection fusion module design. In this paper, we propose Elite360D, a
novel framework that inputs the ERP image and icosahedron projection (ICOSAP)
point set, which is undistorted and spatially continuous. Elite360D is superior
in its capacity in learning a representation from a local-with-global
perspective. With a flexible ERP image encoder, it includes an ICOSAP point
encoder, and a Bi-projection Bi-attention Fusion (B2F) module (totally ~1M
parameters). Specifically, the ERP image encoder can take various perspective
image-trained backbones (e.g., ResNet, Transformer) to extract local features.
The point encoder extracts the global features from the ICOSAP. Then, the B2F
module captures the semantic- and distance-aware dependencies between each
pixel of the ERP feature and the entire ICOSAP feature set. Without specific
backbone design and obvious computational cost increase, Elite360D outperforms
the prior arts on several benchmark datasets.