Hierarchical Neural Memory Network for Low Latency Event Processing
Ryuhei Hamaguchi Yasutaka Furukawa Masaki Onishi Ken Sakurada
This paper proposes a low latency neural network architecture for event-based
dense prediction tasks. Conventional architectures encode entire scene contents
at a fixed rate regardless of their temporal characteristics. Instead, the
proposed network encodes contents at a proper temporal scale depending on its
movement speed. We achieve this by constructing temporal hierarchy using
stacked latent memories that operate at different rates. Given low latency
event steams, the multi-level memories gradually extract dynamic to static
scene contents by propagating information from the fast to the slow memory
modules. The architecture not only reduces the redundancy of conventional
architectures but also exploits long-term dependencies. Furthermore, an
attention-based event representation efficiently encodes sparse event streams
into the memory cells. We conduct extensive evaluations on three event-based
dense prediction tasks, where the proposed approach outperforms the existing
methods on accuracy and latency, while demonstrating effective event and image
fusion capabilities. The code is available at https://hamarh.github.io/hmnet/