Replicating Complex Dialogue Policy of Humans via Offline Imitation
  Learning with Supervised Regularization
Zhoujian Sun Chenyang Zhao Zhengxing Huang Nai Ding
Policy learning (PL) is a module of a task-oriented dialogue system that
trains an agent to make actions in each dialogue turn. Imitating human action
is a fundamental problem of PL. However, both supervised learning (SL) and
reinforcement learning (RL) frameworks cannot imitate humans well. Training RL
models require online interactions with user simulators, while simulating
complex human policy is hard. Performances of SL-based models are restricted
because of the covariate shift problem. Specifically, a dialogue is a
sequential decision-making process where slight differences in current
utterances and actions will cause significant differences in subsequent
utterances. Therefore, the generalize ability of SL models is restricted
because statistical characteristics of training and testing dialogue data
gradually become different. This study proposed an offline imitation learning
model that learns policy from real dialogue datasets and does not require user
simulators. It also utilizes state transition information, which alleviates the
influence of the covariate shift problem. We introduced a regularization trick
to make our model can be effectively optimized. We investigated the performance
of our model on four independent public dialogue datasets. The experimental
result showed that our model performed better in the action prediction task.