Modality-Aware Representation Learning for Zero-shot Sketch-based Image
  Retrieval
Eunyi Lyou Doyeon Lee Jooeun Kim Joonseok Lee
Zero-shot learning offers an efficient solution for a machine learning model
to treat unseen categories, avoiding exhaustive data collection. Zero-shot
Sketch-based Image Retrieval (ZS-SBIR) simulates real-world scenarios where it
is hard and costly to collect paired sketch-photo samples. We propose a novel
framework that indirectly aligns sketches and photos by contrasting them
through texts, removing the necessity of access to sketch-photo pairs. With an
explicit modality encoding learned from data, our approach disentangles
modality-agnostic semantics from modality-specific information, bridging the
modality gap and enabling effective cross-modal content retrieval within a
joint latent space. From comprehensive experiments, we verify the efficacy of
the proposed model on ZS-SBIR, and it can be also applied to generalized and
fine-grained settings.