Fusing Hand and Body Skeletons for Human Action Recognition in Assembly
Dustin Aganian Mona Köhler Benedict Stephan Markus Eisenbach Horst-Michael Gross
As collaborative robots (cobots) continue to gain popularity in industrial
manufacturing, effective human-robot collaboration becomes crucial. Cobots
should be able to recognize human actions to assist with assembly tasks and act
autonomously. To achieve this, skeleton-based approaches are often used due to
their ability to generalize across various people and environments. Although
body skeleton approaches are widely used for action recognition, they may not
be accurate enough for assembly actions where the worker's fingers and hands
play a significant role. To address this limitation, we propose a method in
which less detailed body skeletons are combined with highly detailed hand
skeletons. We investigate CNNs and transformers, the latter of which are
particularly adept at extracting and combining important information from both
skeleton types using attention. This paper demonstrates the effectiveness of
our proposed approach in enhancing action recognition in assembly scenarios.