Efficient Gesture Recognition on Spiking Convolutional Networks Through
  Sensor Fusion of Event-Based and Depth Data
Lea Steffen Thomas Trapp Arne Roennau Rüdiger Dillmann
As intelligent systems become increasingly important in our daily lives, new
ways of interaction are needed. Classical user interfaces pose issues for the
physically impaired and are partially not practical or convenient. Gesture
recognition is an alternative, but often not reactive enough when conventional
cameras are used. This work proposes a Spiking Convolutional Neural Network,
processing event- and depth data for gesture recognition. The network is
simulated using the open-source neuromorphic computing framework LAVA for
offline training and evaluation on an embedded system. For the evaluation three
open source data sets are used. Since these do not represent the applied
bi-modality, a new data set with synchronized event- and depth data was
recorded. The results show the viability of temporal encoding on depth
information and modality fusion, even on differently encoded data, to be
beneficial to network performance and generalization capabilities.