Learning Part Segmentation from Synthetic Animals
Jiawei Peng Ju He Prakhar Kaushik Zihao Xiao Jiteng Mu Alan Yuille
Semantic part segmentation provides an intricate and interpretable
understanding of an object, thereby benefiting numerous downstream tasks.
However, the need for exhaustive annotations impedes its usage across diverse
object types. This paper focuses on learning part segmentation from synthetic
animals, leveraging the Skinned Multi-Animal Linear (SMAL) models to scale up
existing synthetic data generated by computer-aided design (CAD) animal models.
Compared to CAD models, SMAL models generate data with a wider range of poses
observed in real-world scenarios. As a result, our first contribution is to
construct a synthetic animal dataset of tigers and horses with more pose
diversity, termed Synthetic Animal Parts (SAP). We then benchmark Syn-to-Real
animal part segmentation from SAP to PartImageNet, namely SynRealPart, with
existing semantic segmentation domain adaptation methods and further improve
them as our second contribution. Concretely, we examine three Syn-to-Real
adaptation methods but observe relative performance drop due to the innate
difference between the two tasks. To address this, we propose a simple yet
effective method called Class-Balanced Fourier Data Mixing (CB-FDM). Fourier
Data Mixing aligns the spectral amplitudes of synthetic images with real
images, thereby making the mixed images have more similar frequency content to
real images. We further use Class-Balanced Pseudo-Label Re-Weighting to
alleviate the imbalanced class distribution. We demonstrate the efficacy of
CB-FDM on SynRealPart over previous methods with significant performance
improvements. Remarkably, our third contribution is to reveal that the learned
parts from synthetic tiger and horse are transferable across all quadrupeds in
PartImageNet, further underscoring the utility and potential applications of
animal part segmentation.