Rescue Conversations from Dead-ends: Efficient Exploration for
  Task-oriented Dialogue Policy Optimization
Yangyang Zhao Zhenyu Wang Mehdi Dastani Shihan Wang
Training a dialogue policy using deep reinforcement learning requires a lot
of exploration of the environment. The amount of wasted invalid exploration
makes their learning inefficient. In this paper, we find and define an
important reason for the invalid exploration: dead-ends. When a conversation
enters a dead-end state, regardless of the actions taken afterward, it will
continue in a dead-end trajectory until the agent reaches a termination state
or maximum turn. We propose a dead-end resurrection (DDR) algorithm that
detects the initial dead-end state in a timely and efficient manner and
provides a rescue action to guide and correct the exploration direction. To
prevent dialogue policies from repeatedly making the same mistake, DDR also
performs dialogue data augmentation by adding relevant experiences containing
dead-end states. We first validate the dead-end detection reliability and then
demonstrate the effectiveness and generality of the method by reporting
experimental results on several dialogue datasets from different domains.