Multi-goal Audio-visual Navigation using Sound Direction Map
Haru Kondoh Asako Kanezaki
Over the past few years, there has been a great deal of research on
navigation tasks in indoor environments using deep reinforcement learning
agents. Most of these tasks use only visual information in the form of
first-person images to navigate to a single goal. More recently, tasks that
simultaneously use visual and auditory information to navigate to the sound
source and even navigation tasks with multiple goals instead of one have been
proposed. However, there has been no proposal for a generalized navigation task
combining these two types of tasks and using both visual and auditory
information in a situation where multiple sound sources are goals. In this
paper, we propose a new framework for this generalized task: multi-goal
audio-visual navigation. We first define the task in detail, and then we
investigate the difficulty of the multi-goal audio-visual navigation task
relative to the current navigation tasks by conducting experiments in various
situations. The research shows that multi-goal audio-visual navigation has the
difficulty of the implicit need to separate the sources of sound. Next, to
mitigate the difficulties in this new task, we propose a method named sound
direction map (SDM), which dynamically localizes multiple sound sources in a
learning-based manner while making use of past memories. Experimental results
show that the use of SDM significantly improves the performance of multiple
baseline methods, regardless of the number of goals.