SketchBodyNet: A Sketch-Driven Multi-faceted Decoder Network for 3D
  Human Reconstruction
Fei Wang Kongzhang Tang Hefeng Wu Baoquan Zhao Hao Cai Teng Zhou
Reconstructing 3D human shapes from 2D images has received increasing
attention recently due to its fundamental support for many high-level 3D
applications. Compared with natural images, freehand sketches are much more
flexible to depict various shapes, providing a high potential and valuable way
for 3D human reconstruction. However, such a task is highly challenging. The
sparse abstract characteristics of sketches add severe difficulties, such as
arbitrariness, inaccuracy, and lacking image details, to the already badly
ill-posed problem of 2D-to-3D reconstruction. Although current methods have
achieved great success in reconstructing 3D human bodies from a single-view
image, they do not work well on freehand sketches. In this paper, we propose a
novel sketch-driven multi-faceted decoder network termed SketchBodyNet to
address this task. Specifically, the network consists of a backbone and three
separate attention decoder branches, where a multi-head self-attention module
is exploited in each decoder to obtain enhanced features, followed by a
multi-layer perceptron. The multi-faceted decoders aim to predict the camera,
shape, and pose parameters, respectively, which are then associated with the
SMPL model to reconstruct the corresponding 3D human mesh. In learning,
existing 3D meshes are projected via the camera parameters into 2D synthetic
sketches with joints, which are combined with the freehand sketches to optimize
the model. To verify our method, we collect a large-scale dataset of about 26k
freehand sketches and their corresponding 3D meshes containing various poses of
human bodies from 14 different angles. Extensive experimental results
demonstrate our SketchBodyNet achieves superior performance in reconstructing
3D human meshes from freehand sketches.