Multi-task Learning with 3D-Aware Regularization
Wei-Hong Li Steven McDonagh Ales Leonardis Hakan Bilen
Deep neural networks have become a standard building block for designing
models that can perform multiple dense computer vision tasks such as depth
estimation and semantic segmentation thanks to their ability to capture complex
correlations in high dimensional feature space across tasks. However, the
cross-task correlations that are learned in the unstructured feature space can
be extremely noisy and susceptible to overfitting, consequently hurting
performance. We propose to address this problem by introducing a structured
3D-aware regularizer which interfaces multiple tasks through the projection of
features extracted from an image encoder to a shared 3D feature space and
decodes them into their task output space through differentiable rendering. We
show that the proposed method is architecture agnostic and can be plugged into
various prior multi-task backbones to improve their performance; as we evidence
using standard benchmarks NYUv2 and PASCAL-Context.