Mainstream captioning models often follow a sequential structure to generate cap-
 tions, leading to issues such as introduction of irrelevant semantics, lack of diversity
 in the generated captions, and inadequate generalization performance. In this paper,
 we present an alternative paradigm for image captioning, which factorizes the
 captioning procedure into two stages: (1) extracting an explicit semantic represen-
 tation from the given image; and (2) constructing the caption based on a recursive
 compositional procedure in a bottom-up manner. Compared to conventional ones,
 our paradigm better preserves the semantic content through an explicit factorization
 of semantics and syntax. By using the compositional generation procedure, caption
 construction follows a recursive structure, which naturally fits the properties of
 human language. Moreover, the proposed compositional procedure requires less
 data to train, generalizes better, and yields more diverse captions. Image captioning, the task to generate short descriptions for given images, has received increasing attention in recent years. State-of-the-art models BID0 BID1 BID2 BID3 mostly adopt the encoder-decoder paradigm BID2 , where the content of the given image is first encoded via a convolutional network into a feature vector, which is then decoded into a caption via a recurrent network. In particular, the words in the caption are produced in a sequential manner -the choice of each word depends on both the preceding word and the image feature. Despite its simplicity and the effectiveness shown on various benchmarks BID4 BID5 , the sequential model has a fundamental problem. Specifically, it could not reflect the inherent hierarchical structures of natural languages BID6 BID7 in image captioning and other generation tasks, although it could implicitly capture such structures in tasks taking the complete sentences as input, e.g. parsing BID8 , and classification BID9 .As a result, sequential models have several significant drawbacks. First , they rely excessively on n-gram statistics rather than hierarchical dependencies among words in a caption. Second , such models usually favor the frequent n-grams BID10 in the training set, which, as shown in Figure 1 , may lead to captions that are only correct syntactically but not semantically, containing semantic concepts that are irrelevant to the conditioned image. Third , the entanglement of syntactic rules and semantics obscures the dependency structure and makes sequential models difficult to generalize.To tackle these issues, we propose a new paradigm for image captioning, where the extraction of semantics (i.e. what to say) and the construction of syntactically correct captions (i.e. how to say) are decomposed into two stages. Specifically , it derives an explicit representation of the semantic content of the given image, which comprises a set of noun-phrases, e.g. a white cat, a cloudy sky or two men. With these noun-phrases as the basis, it then proceeds to construct the caption through recursive composition until a complete caption is obtained. In particular , at each step of the composition, a higher-level phrase is formed by joining two selected sub-phrases via a connecting phrase. It is Preprint . Work in progress.a large building with a clock tower a building with a clock on the side of it a building with a clock on the side of it Figure 1 : This figure shows three test images in MS-COCO BID4 with captions generated by the neural image captioner BID2 , which contain n-gram building with a clock that appeared frequently in the training set but is not semantically correct for these images.noteworthy that the compositional procedure described above is not a hand-crafted algorithm. Instead, it consists of two parametric modular nets, a connecting module for phrase composition and an evaluation module for deciding the completeness of phrases.The proposed paradigm has several key advantages compared to conventional captioning models: BID0 The factorization of semantics and syntax not only better preserves the semantic content of the given image but also makes caption generation easy to interpret and control. (2) The recursive composition procedure naturally reflects the inherent structures of natural language and allows the hierarchical dependencies among words and phrases to be captured. Through a series of ablative studies, we show that the proposed paradigm can effectively increase the diversity of the generated captions while preserving semantic correctness. It also generalizes better to new data and can maintain reasonably good performance when the number of available training data is small. In this paper, we propose a novel paradigm for image captioning. While the typical existing approaches encode images using feature vectors and generate captions sequentially, the proposed method generates captions in a compositional manner. In particular, our approach factorizes the captioning procedure into two stages. In the first stage, an explicit representation of the input image, consisting of noun-phrases, is extracted. In the second stage, a recursive compositional procedure is applied to assemble extracted noun-phrases into a caption. As a result, caption generation follows a hierarchical structure, which naturally fits the properties of human language. On two datasets, the proposed compositional procedure is shown to preserve semantics more effectively, require less data to train, generalize better across datasets, and yield more diverse captions.
a hierarchical and compositional way to generate captions This paper presents a more interpretable method for image captioning.