Emergent Communication in Interactive Sketch Question Answering
Zixing Lei Yiming Zhang Yuxin Xiong Siheng Chen
Vision-based emergent communication (EC) aims to learn to communicate through
sketches and demystify the evolution of human communication. Ironically,
previous works neglect multi-round interaction, which is indispensable in human
communication. To fill this gap, we first introduce a novel Interactive Sketch
Question Answering (ISQA) task, where two collaborative players are interacting
through sketches to answer a question about an image in a multi-round manner.
To accomplish this task, we design a new and efficient interactive EC system,
which can achieve an effective balance among three evaluation factors,
including the question answering accuracy, drawing complexity and human
interpretability. Our experimental results including human evaluation
demonstrate that multi-round interactive mechanism facilitates targeted and
efficient communication between intelligent agents with decent human
interpretability.