Grounding and Distinguishing Conceptual Vocabulary Through Similarity
  Learning in Embodied Simulations
Sadaf Ghaffari Nikhil Krishnaswamy
We present a novel method for using agent experiences gathered through an
embodied simulation to ground contextualized word vectors to object
representations. We use similarity learning to make comparisons between
different object types based on their properties when interacted with, and to
extract common features pertaining to the objects' behavior. We then use an
affine transformation to calculate a projection matrix that transforms
contextualized word vectors from different transformer-based language models
into this learned space, and evaluate whether new test instances of transformed
token vectors identify the correct concept in the object embedding space. Our
results expose properties of the embedding spaces of four different transformer
models and show that grounding object token vectors is usually more helpful to
grounding verb and attribute token vectors than the reverse, which reflects
earlier conclusions in the analogical reasoning and psycholinguistic
literature.