Exploring Failure Cases in Multimodal Reasoning About Physical Dynamics
Sadaf Ghaffari Nikhil Krishnaswamy
In this paper, we present an exploration of LLMs' abilities to problem solve
with physical reasoning in situated environments. We construct a simple
simulated environment and demonstrate examples of where, in a zero-shot
setting, both text and multimodal LLMs display atomic world knowledge about
various objects but fail to compose this knowledge in correct solutions for an
object manipulation and placement task. We also use BLIP, a vision-language
model trained with more sophisticated cross-modal attention, to identify cases
relevant to object physical properties that that model fails to ground.
Finally, we present a procedure for discovering the relevant properties of
objects in the environment and propose a method to distill this knowledge back
into the LLM.