Neural Slot Interpreters: Grounding Object Semantics in Emergent Slot
  Representations
Bhishma Dedhia Niraj K. Jha
Object-centric methods have seen significant progress in unsupervised
decomposition of raw perception into rich object-like abstractions. However,
limited ability to ground object semantics of the real world into the learned
abstractions has hindered their adoption in downstream understanding
applications. We present the Neural Slot Interpreter (NSI) that learns to
ground and generate object semantics via slot representations. At the core of
NSI is an XML-like programming language that uses simple syntax rules to
organize the object semantics of a scene into object-centric program
primitives. Then, an alignment model learns to ground program primitives into
slots through a bi-level contrastive learning objective over a shared embedding
space. Finally, we formulate the NSI program generator model to use the dense
associations inferred from the alignment model to generate object-centric
programs from slots. Experiments on bi-modal retrieval tasks demonstrate the
efficacy of the learned alignments, surpassing set-matching-based predictors by
a significant margin. Moreover, learning the program generator from grounded
associations enhances the predictive power of slots. NSI generated programs
demonstrate improved performance of object-centric learners on property
prediction and object detection, and scale with real-world scene complexity.