Label-Free Event-based Object Recognition via Joint Learning with Image
  Reconstruction from Events
Hoonhee Cho Hyeonseong Kim Yujeong Chae Kuk-Jin Yoon
Recognizing objects from sparse and noisy events becomes extremely difficult
when paired images and category labels do not exist. In this paper, we study
label-free event-based object recognition where category labels and paired
images are not available. To this end, we propose a joint formulation of object
recognition and image reconstruction in a complementary manner. Our method
first reconstructs images from events and performs object recognition through
Contrastive Language-Image Pre-training (CLIP), enabling better recognition
through a rich context of images. Since the category information is essential
in reconstructing images, we propose category-guided attraction loss and
category-agnostic repulsion loss to bridge the textual features of predicted
categories and the visual features of reconstructed images using CLIP.
Moreover, we introduce a reliable data sampling strategy and local-global
reconstruction consistency to boost joint learning of two tasks. To enhance the
accuracy of prediction and quality of reconstruction, we also propose a
prototype-based approach using unpaired images. Extensive experiments
demonstrate the superiority of our method and its extensibility for zero-shot
object recognition. Our project code is available at
\url{https://github.com/Chohoonhee/Ev-LaFOR}.