Training with Product Digital Twins for AutoRetail Checkout
Yue Yao Xinyu Tian Zheng Tang Sujit Biswas Huan Lei Tom Gedeon Liang Zheng
Automating the checkout process is important in smart retail, where users
effortlessly pass products by hand through a camera, triggering automatic
product detection, tracking, and counting. In this emerging area, due to the
lack of annotated training data, we introduce a dataset comprised of product 3D
models, which allows for fast, flexible, and large-scale training data
generation through graphic engine rendering. Within this context, we discern an
intriguing facet, because of the user "hands-on" approach, bias in user
behavior leads to distinct patterns in the real checkout process. The existence
of such patterns would compromise training effectiveness if training data fail
to reflect the same. To address this user bias problem, we propose a training
data optimization framework, i.e., training with digital twins (DtTrain).
Specifically, we leverage the product 3D models and optimize their rendering
viewpoint and illumination to generate "digital twins" that visually resemble
representative user images. These digital twins, inherit product labels and,
when augmented, form the Digital Twin training set (DT set). Because the
digital twins individually mimic user bias, the resulting DT training set
better reflects the characteristics of the target scenario and allows us to
train more effective product detection and tracking models. In our experiment,
we show that DT set outperforms training sets created by existing dataset
synthesis methods in terms of counting accuracy. Moreover, by combining DT set
with pseudo-labeled real checkout data, further improvement is observed. The
code is available at https://github.com/yorkeyao/Automated-Retail-Checkout.