How Object Information Improves Skeleton-based Human Action Recognition
  in Assembly Tasks
Dustin Aganian Mona Köhler Sebastian Baake Markus Eisenbach Horst-Michael Gross
As the use of collaborative robots (cobots) in industrial manufacturing
continues to grow, human action recognition for effective human-robot
collaboration becomes increasingly important. This ability is crucial for
cobots to act autonomously and assist in assembly tasks. Recently,
skeleton-based approaches are often used as they tend to generalize better to
different people and environments. However, when processing skeletons alone,
information about the objects a human interacts with is lost. Therefore, we
present a novel approach of integrating object information into skeleton-based
action recognition. We enhance two state-of-the-art methods by treating object
centers as further skeleton joints. Our experiments on the assembly dataset
IKEA ASM show that our approach improves the performance of these
state-of-the-art methods to a large extent when combining skeleton joints with
objects predicted by a state-of-the-art instance segmentation model. Our
research sheds light on the benefits of combining skeleton joints with object
information for human action recognition in assembly tasks. We analyze the
effect of the object detector on the combination for action classification and
discuss the important factors that must be taken into account.