To Supervise or Not to Supervise: Understanding and Addressing the Key
  Challenges of 3D Transfer Learning
Souhail Hadgi Lei Li Maks Ovsjanikov
Transfer learning has long been a key factor in the advancement of many
fields including 2D image analysis. Unfortunately, its applicability in 3D data
processing has been relatively limited. While several approaches for 3D
transfer learning have been proposed in recent literature, with contrastive
learning gaining particular prominence, most existing methods in this domain
have only been studied and evaluated in limited scenarios. Most importantly,
there is currently a lack of principled understanding of both when and why 3D
transfer learning methods are applicable. Remarkably, even the applicability of
standard supervised pre-training is poorly understood. In this work, we conduct
the first in-depth quantitative and qualitative investigation of supervised and
contrastive pre-training strategies and their utility in downstream 3D tasks.
We demonstrate that layer-wise analysis of learned features provides
significant insight into the downstream utility of trained networks. Informed
by this analysis, we propose a simple geometric regularization strategy, which
improves the transferability of supervised pre-training. Our work thus sheds
light onto both the specific challenges of 3D transfer learning, as well as
strategies to overcome them.