Learning Keypoints for Robotic Cloth Manipulation using Synthetic Data
Thomas Lips Victor-Louis De Gusseme Francis wyffels
Assistive robots should be able to wash, fold or iron clothes. However, due
to the variety, deformability and self-occlusions of clothes, creating
general-purpose robot systems for cloth manipulation is challenging. Synthetic
data is a promising direction to improve generalization, though its usability
is often limited by the sim-to-real gap. To advance the use of synthetic data
for cloth manipulation and to enable tasks such as robotic folding, we present
a synthetic data pipeline to train keypoint detectors for almost flattened
cloth items. To test its performance, we have also collected a real-world
dataset. We train detectors for both T-shirts, towels and shorts and obtain an
average precision of 64.3%. Fine-tuning on real-world data improves performance
to 74.2%. Additional insight is provided by discussing various failure modes of
the keypoint detectors and by comparing different approaches to obtain cloth
meshes and materials. We also quantify the remaining sim-to-real gap and argue
that further improvements to the fidelity of cloth assets will be required to
further reduce this gap. The code, dataset and trained models are available
online.