Learning to Summarize and Answer Questions about a Virtual Robot's Past
  Actions
Chad DeChant Iretiayo Akinola Daniel Bauer
When robots perform long action sequences, users will want to easily and
reliably find out what they have done. We therefore demonstrate the task of
learning to summarize and answer questions about a robot agent's past actions
using natural language alone. A single system with a large language model at
its core is trained to both summarize and answer questions about action
sequences given ego-centric video frames of a virtual robot and a question
prompt. To enable training of question answering, we develop a method to
automatically generate English-language questions and answers about objects,
actions, and the temporal order in which actions occurred during episodes of
robot action in the virtual environment. Training one model to both summarize
and answer questions enables zero-shot transfer of representations of objects
learned through question answering to improved action summarization. %
involving objects not seen in training to summarize.