Despite much success in many large-scale language tasks, sequence-to-sequence (seq2seq) models have not been an ideal choice for conversational modeling as they tend to generate generic and repetitive responses. In this paper, we propose a Latent Topic Conversational Model (LTCM) that augments the seq2seq model with a neural topic component to better model human-human conversations. The neural topic component encodes information from the source sentence to build a global “topic” distribution over words, which is then consulted by the seq2seq model to improve generation at each time step. The experimental results show that the proposed LTCM can generate more diverse and interesting responses by sampling from its learnt latent representations. In a subjective human evaluation, the judges also confirm that LTCM is the preferred option comparing to competitive baseline models.
 Sequence-to-Sequence model (seq2seq) BID35 , as a data-driven approach to mapping between two arbitrary length sequences, has attracted much attention and been widely applied to many natural language processing tasks such as machine translation BID20 , syntactic parsing , and summarisation BID28 . Neural conversational models BID32 BID30 are the latest development in open-domain conversational modelling, where seq2seq-based models are employed for learning dialogue decisions in an end-to-end fashion. Despite promising results, the lack of explicit knowledge representations (or the inability to learn them from data) impedes the model from generating causal or even rational responses. This leads to many problems discussed in previous literature such as generic responses BID17 , inconsistency BID18 , and redundancy and contradiction BID33 .On the other hand, goal-oriented dialogues BID45 use the notion of dialogue ontology to constrain the scope of conversation and facilitate rational system behaviour within the domain. Neural network-based task-oriented dialogue systems usually retrieve knowledge from a pre-defined database either by discrete accessing BID41 BID3 or through an attention mechanism BID8 . The provision of this database offers a proxy for language grounding, which is crucial to guide the generation or selection of the system responses. As shown in BID40 , a stochastic neural dialogue model can generate diverse yet rational responses mainly because they are heavily driven by the knowledge the model is conditioned on.Despite the need for explicit knowledge representations, building a general-purpose knowledge base and actually making use of it have been proven difficult BID22 BID25 . Therefore, progress has been made in conditioning the seq2seq model on coarse-grained knowledge representations, such as a fuzzily-matched retrieval result via attention BID10 or a set of pre-organised topic or scenario labels . In this work, we propose a hybrid of a seq2seq conversational model and a neural topic model -Latent Topic Conversational Model (LTCM) -to jointly learn the useful latent representations and the way to make use of them in a conversation. LTCM uses its underlying seq2seq model to capture the local dynamics of a sentence while extracts and represents its global semantics by a mixture of topic components like topic models BID2 . This separation of global semantics and local dynamics turns out to be crucial to the success of LTCM.Recent advances in neural variational inference BID27 BID23 have sparked a series of latent variable models applied to conversational modeling BID31 BID5 . The majority of the work passes a Gaussian random variable to the hidden state of the LSTM decoder and employs the reparameterisation trick BID15 to build an unbiased and low-variance gradient estimator for updating the model parameters. However, studies have shown that training this type of models for language generation tasks is tough because the effect of the latent variable tends to vanish and the language model would take over the entire generation process over time BID4 . This results in several workarounds such as KL annealing BID4 BID5 , word dropout and historyless decoding BID4 , as well as auxiliary bag-of-word signals . Unlike previous approaches, LTCM is similar to TopicRNN (Dieng et al., 2017) where it passes the latent variable to the output layer of the decoder and only back-propagates the gradient of the topic words to the latent variable.In summary, the contribution of this paper is two-fold: first and most importantly, we show that LTCM can learn to generate more diverse and interesting responses by sampling from the learnt topic representations. The results were confirmed by a corpus-based evaluation and a human assessment; secondly, we conducted a series of experiments to understand the properties of seq2seq-based latent variables models better, which may serve as rules of thumb for future model development. In this paper, we have proposed the Latent Topic Conversational Model (LTCM) for general-purpose conversational modeling. We have shown that LTCM can generate more interesting and diverse responses by combining the seq2seq model and neural topic model so that global semantic representations and local word transitions can be modeled separately but learned jointly. Both a corpus-based evaluation and a human assessment confirm this finding. Future work would be to study the learned representations and use them to control the meaning of the generated responses.
Latent Topic Conversational Model, a hybrid of seq2seq and neural topic model to generate more diverse and interesting responses. This paper proposed the combination of topic model and seq2seq conversational model Proposes a conversational model with topical information by combining seq2seq model with neural topic models and shows the proposed model outperforms some the baseline model seq2seq and other latent variable model variant of seq2seq. The paper addresses the issue of enduring topicality in conversation models and proposes a model which is a combination of a neural topic model and a seq2seq-based dialog system. 