Towards Balanced RGB-TSDF Fusion for Consistent Semantic Scene
  Completion by 3D RGB Feature Completion and a Classwise Entropy Loss Function
Laiyan Ding Panwen Hu Jie Li Rui Huang
Semantic Scene Completion (SSC) aims to jointly infer semantics and
occupancies of 3D scenes. Truncated Signed Distance Function (TSDF), a 3D
encoding of depth, has been a common input for SSC. Furthermore, RGB-TSDF
fusion, seems promising since these two modalities provide color and geometry
information, respectively. Nevertheless, RGB-TSDF fusion has been considered
nontrivial and commonly-used naive addition will result in inconsistent
results. We argue that the inconsistency comes from the sparsity of RGB
features upon projecting into 3D space, while TSDF features are dense, leading
to imbalanced feature maps when summed up. To address this RGB-TSDF
distribution difference, we propose a two-stage network with a 3D RGB feature
completion module that completes RGB features with meaningful values for
occluded areas. Moreover, we propose an effective classwise entropy loss
function to punish inconsistency. Extensive experiments on public datasets
verify that our method achieves state-of-the-art performance among methods that
do not adopt extra data.