Context information plays an important role in human language understanding, and it is also useful for machines to learn vector representations of language. In this paper, we explore an asymmetric encoder-decoder structure for unsupervised context-based sentence representation learning. As a result, we build an encoder-decoder architecture with an RNN encoder and a CNN decoder, and we show that neither an autoregressive decoder nor an RNN decoder is required.   We further combine a suite of effective designs to significantly improve model efficiency while also achieving better performance. Our model is trained on two different large unlabeled corpora, and in both cases transferability is evaluated on a set of downstream language understanding tasks. We empirically show that our model is simple and fast while producing rich sentence representations that excel in downstream tasks. Learning distributed representations of sentences is an important and hard topic in both the deep learning and natural language processing communities, since it requires machines to encode a sentence with rich language content into a fixed-dimension vector filled with continuous values. We are interested in learning to build a distributed sentence encoder in an unsupervised fashion by exploiting the structure and relationship in a large unlabeled corpus. Since humans interpret sentences by composing from the meanings of the words, we decompose the task of learning a sentence encoder into two essential components: learning distributed word representations, and learning how to compose a sentence representation from the representations of words in the given sentence.Numerous studies in human language processing have claimed that the context in which words and sentences are understood plays an important role in human language understanding BID1 BID4 . The idea of learning from the context information BID35 was recently successfully applied to vector representation learning for words in ; BID30 . BID8 proposed a unified framework for learning language representation from the unlabeled data, and it is able to generalize to various NLP tasks. Inspired by the prior work on incorporating context information into representation learning, proposed the Skipthought model, which is an encoder-decoder model for unsupervised sentence representation learning. The paper exploits the semantic similarity within a tuple of adjacent sentences as supervision, and successfully built a generic, distributed sentence encoder. Rather than applying the conventional autoencoder model, the skip-thought model tries to reconstruct the surrounding 2 sentences instead of the input sentence. The learned sentence representation encoder outperforms previous unsupervised pretrained models on the evaluation tasks with no finetuning, and the results are comparable to the models which were trained directly on the datasets in a supervised fashion.The usage of 2 independent decoders in Skip-thought model matches our intuition that, given the current sentence, inferring the previous sentence and inferring the next one should be different.Figure 1: Our proposed model is composed of an RNN encoder, and a CNN decoder. During training, a batch of sentences are sent to the model, and the RNN encoder computes a vector representation for each of sentences; then the CNN decoder needs to reconstruct the paired target sequence, which contains 30 contiguous words right after the input sentence, given the vector representation. 300 is the dimension of word vectors. D is the dimension of sentence representation, and it varies along with the change of the RNN encoder size. (Better view in color.) DISPLAYFORM0 Representation: We aim to provide a model with faster training speed with better transferability than existing algorithms, thus we choose to apply a parameter-free composition function, which is a concatenation of the outputs from a global mean pooling over time and a global max pooling over time, on the computed sequence of hidden states. The composition function can be represented as DISPLAYFORM1 where max H d· is the max operation on the d-th row of the matrix H, which outputs a scalar. Thus the representation z has a dimension of 2d.Decoder: The decoder is a 3-layer CNN to reconstruct the paired target sequence t, which needs to expand z from length 1 to the length of t. Intuitively, the decoder could be a stack of deconvolution layers. For fast training speed, we optimized the architecture to make it plausible to use fullyconnected layers and convolution layers in the decoder, since generally, convolution layers run faster than deconvolution layers in modern deep learning frameworks.Suppose that the target sequence t has N words, the first layer of deconvolution will expand z, which could be considered as a sequence with length 1, into a feature map with length N . It can be easily implemented as a concatenation of outputs from N linear transformations in parallel. Then the second and third layer are 1D-convolution layers with kernel size 3 and 1, respectively. The output feature DISPLAYFORM2 , where v ∈ R e , and e is dimension of the word vectors.Note that our decoder is not an autoregressive model, and it brings us high training efficiency. We will discuss the reason of choosing this decoder which we call a predict-all-words CNN decoder.Objective: A softmax layer is applied after the decoder to produce a probability distribution over words at each position, softmax(Ev n ), and the training objective is to minimize the sum of the negative log-likelihood over all positions in the target sequence t: DISPLAYFORM3 The loss function L is summed over all sentences in the training corpus. Inspired by learning to exploit the contextual information present in adjacent sentences, we proposed an asymmetric encoder-decoder model with a suite of techniques for improving context-based unsupervised sentence representation learning. Since we believe that a simple model will be faster in training and easier to analyze, we opt to use simple techniques in our proposed model, including 1) an RNN as the encoder, and a predict-all-words CNN as the decoder, 2) learning by inferring next contiguous words, 3) mean+max pooling, and 4) tying word vectors with word prediction. With thorough discussion and extensive evaluation, we justify our decision making for each component in our RNN-CNN model. In terms of the performance and the efficiency of training, we justify that our model is a fast and simple algorithm for learning generic sentence representations from unlabeled corpora. Further research will focus on how to maximize the utility of the context information, and how to design simple architectures to best make use of it. 9 , and 12) works better than other asymmetric models (CNN-LSTM, row 11), and models with symmetric structure (RNN-RNN, row 5 and 10). In addition, with larger encoder size, our model demonstrates stronger transferability. The default setting for our CNN decoder is that it learns to reconstruct 30 words right next to every input sentence. "CNN(10)" represents a CNN decoder with the length of outputs as 10, and "CNN(50)" represents it with the length of outputs as 50. " †" indicates that the CNN decoder learns to reconstruct next sentence. " ‡" indicates the results reported in Gan et al. as future predictor. The CNN encoder in our experiment, noted as " §", was based on AdaSent in Zhao et al. and Conneau et al.. Bold numbers are best results among models at same dimension, and underlined numbers are best results among all models. For STS14, the performance measures are Pearson's and Spearman's score. For MSRP, the performance measures are accuracy and F1 score.
We proposed an RNN-CNN encoder-decoder model for fast unsupervised sentence representation learning. Modifications to the skip-thought framework for learning sentence embeddings. This paper presents a new RNN encoder–CNN decoder hybrid design for use in pretraining, which does not require an autoregressive decoder when pretraining encoders. The authors extend Skip-thought by decoding only one target sentence using a CNN decoder.