"Don't forget to put the milk back!" Dataset for Enabling Embodied
  Agents to Detect Anomalous Situations
James F. Mullen Jr Prasoon Goyal Robinson Piramuthu Michael Johnston Dinesh Manocha Reza Ghanadan
Home robots intend to make their users lives easier. Our work assists in this
goal by enabling robots to inform their users of dangerous or unsanitary
anomalies in their home. Some examples of these anomalies include the user
leaving their milk out, forgetting to turn off the stove, or leaving poison
accessible to children. To move towards enabling home robots with these
abilities, we have created a new dataset, which we call SafetyDetect. The
SafetyDetect dataset consists of 1000 anomalous home scenes, each of which
contains unsafe or unsanitary situations for an agent to detect. Our approach
utilizes large language models (LLMs) alongside both a graph representation of
the scene and the relationships between the objects in the scene. Our key
insight is that this connected scene graph and the object relationships it
encodes enables the LLM to better reason about the scene -- especially as it
relates to detecting dangerous or unsanitary situations. Our most promising
approach utilizes GPT-4 and pursues a categorization technique where object
relations from the scene graph are classified as normal, dangerous, unsanitary,
or dangerous for children. This method is able to correctly identify over 90%
of anomalous scenarios in the SafetyDetect Dataset. Additionally, we conduct
real world experiments on a ClearPath TurtleBot where we generate a scene graph
from visuals of the real world scene, and run our approach with no
modification. This setup resulted in little performance loss. The SafetyDetect
Dataset and code will be released to the public upon this papers publication.