ADU-Depth: Attention-based Distillation with Uncertainty Modeling for
  Depth Estimation
Zizhang Wu Zhuozheng Li Zhi-Gang Fan Yunzhe Wu Xiaoquan Wang Rui Tang Jian Pu
Monocular depth estimation is challenging due to its inherent ambiguity and
ill-posed nature, yet it is quite important to many applications. While recent
works achieve limited accuracy by designing increasingly complicated networks
to extract features with limited spatial geometric cues from a single RGB
image, we intend to introduce spatial cues by training a teacher network that
leverages left-right image pairs as inputs and transferring the learned 3D
geometry-aware knowledge to the monocular student network. Specifically, we
present a novel knowledge distillation framework, named ADU-Depth, with the
goal of leveraging the well-trained teacher network to guide the learning of
the student network, thus boosting the precise depth estimation with the help
of extra spatial scene information. To enable domain adaptation and ensure
effective and smooth knowledge transfer from teacher to student, we apply both
attention-adapted feature distillation and focal-depth-adapted response
distillation in the training stage. In addition, we explicitly model the
uncertainty of depth estimation to guide distillation in both feature space and
result space to better produce 3D-aware knowledge from monocular observations
and thus enhance the learning for hard-to-predict image regions. Our extensive
experiments on the real depth estimation datasets KITTI and DrivingStereo
demonstrate the effectiveness of the proposed method, which ranked 1st on the
challenging KITTI online benchmark.