Why Guided Dialog Policy Learning performs well? Understanding the role
  of adversarial learning and its alternative
Sho Shimoyama Tetsuro Morimura Kenshi Abe Toda Takamichi Yuta Tomomatsu Masakazu Sugiyama Asahi Hentona Yuuki Azuma Hirotaka Ninomiya
Dialog policies, which determine a system's action based on the current state
at each dialog turn, are crucial to the success of the dialog. In recent years,
reinforcement learning (RL) has emerged as a promising option for dialog policy
learning (DPL). In RL-based DPL, dialog policies are updated according to
rewards. The manual construction of fine-grained rewards, such as
state-action-based ones, to effectively guide the dialog policy is challenging
in multi-domain task-oriented dialog scenarios with numerous state-action pair
combinations. One way to estimate rewards from collected data is to train the
reward estimator and dialog policy simultaneously using adversarial learning
(AL). Although this method has demonstrated superior performance
experimentally, it is fraught with the inherent problems of AL, such as mode
collapse. This paper first identifies the role of AL in DPL through detailed
analyses of the objective functions of dialog policy and reward estimator.
Next, based on these analyses, we propose a method that eliminates AL from
reward estimation and DPL while retaining its advantages. We evaluate our
method using MultiWOZ, a multi-domain task-oriented dialog corpus.