Learning rich and compact representations is an open topic in many fields such as word embedding, visual question-answering, object recognition or image retrieval. Although deep neural networks (convolutional or not) have made a major breakthrough during the last few years by providing hierarchical, semantic and abstract representations for all of these tasks, these representations are not necessary as rich as needed nor as compact as expected. Models using higher order statistics, such as bilinear pooling, provide richer representations at the cost of higher dimensional features. Factorization schemes have been proposed but without being able to reach the original compactness of first order models, or at a heavy loss in performances. This paper addresses these two points by extending factorization schemes to codebook strategies, allowing compact representations with the same dimensionality as first order representations, but with second order performances. Moreover, we extend this framework with a joint codebook and factorization scheme, granting a reduction both in terms of parameters and computation cost. This formulation leads to state-of-the-art results and compact second-order models with few additional parameters and intermediate representations with a dimension similar to that of first-order statistics. Learning rich and compact representations is an open topic in many fields such as word embedding BID16 ), visual question-answering ), object recognition BID23 ) or image retrieval BID19 ). The standard approach extracts features from the input data (text, image, etc.) and builds a representation that will be next processed for a given task (classification, retrieval, etc.) . These features are usually extracted with deep neural networks and the representation is trained in an end-to-end manner. Recently, representations that compute first order statistics over input data have been outperformed by improved models that compute higher order statistics such as bilinear models. This embedding strategy generates richer representations and has been applied in a wide range of tasks : word embedding BID2 ), VQA BID9 ), fine grained classification BID28 ), etc. and gets state-of-the-art results. For instance, Bilinear models perform the best for fine grained visual classification tasks by producing efficient representations that model more details within an image than classical first order statistics BID14 ).However , even if the increase in performances is unquestionable, second order models suffer from a collection of drawbacks: Their intermediate dimension increases quadratically with respect to input features dimension, they require a projection to lower dimension that is costly both in number of parameters and in computation, they are harder to train than first order models due to the increased dimension, they lack a proper adapted pooling scheme which leads to sub-optimal representations.The two main downsides, namely the high dimensional output representations and the sub-efficient pooling scheme, have been widely studied over the last decade. On one hand, the dimensionality issue has been studied through factorization scheme, either representation oriented such as Compact Bilinear Pooling ) and Hadamard Product for Low Rank Bilinear Pooling BID9 ), or task oriented as Low-rank Bilinear Pooling BID10 ). While these factorization schemes are efficient in term of computation cost and number of parameters, the intermediate representation is still too large (typically 10k dimension) to ease the training process and using lower dimension greatly deteriorate performances.On the other hand, it is well-known that global average pooling schemes aggregate unrelated features. This problem has been tackled by the use of codebooks such as VLAD BID0 ) or, in the case of second-order information, Fisher Vectors BID20 ). These strategies have been enhanced to be trainable in an end-to-end manner BID1 ; BID24 ). However , using a codebook on end-to-end trainable second order features leads to an unreasonably large model, since the already large second order model has to be duplicated for each entry of the codebook. This is for example the case in MFAFVNet BID13 ) for which the second order layer alone (i.e., without the CNN part) already costs over 25M parameters and 40 GFLOP, or about as much as an entire ResNet50.In this paper, we tackle both of these shortcomings (intermediate representation cost and lack of proper pooling) by exploring joint factorization and codebook strategies. Our main results are the following:-We first show that state-of-the-art factorization schemes can already be improved by the use of a codebook pooling, albeit at a prohibitive cost. -We then propose our main contribution, a joint codebook and factorization scheme that achieves similar results at a much reduced cost.Since our approach focuses on representation learning and is task agnostic, we validate it in a retrieval context on several image datasets to show the relevance of the learned representations. We show our model achieves competitive results on these datasets at a very reasonable cost.The remaining of this paper is organized as follows: in the next section, we present the related work on second order pooling, factorization schemes and codebook strategies. In section 3, we present our factorization with the codebook strategy and how we improve its integration. In section 4, we show an ablation study on the Stanford Online Products dataset BID18 ). Finally, we compare our approach to the state-of-the-art methods on three image retrieval datasets (Stanford Online Products, CUB-200-2001, Cars-196) . In this paper, we propose a new pooling scheme based which is both efficient in performances (rich representation) and in representation dimension (compact representation). This is thanks to the second-order information that allows richer representation than first-order statistics and thanks to a codebook strategy which pools only related features. To control the computational cost, we extend this pooling scheme with a factorization that shares sets of projections between each entry of the codebook, trading fewer parameters and fewer computation for a small loss in performance. We achieve state-of-the-art results on Stanford Online Products and Cars-196, two image retrieval datasets. Even if our tests are performed on image retrieval datasets, we believe our method can readily be used in place of global average pooling for any task.
We propose a joint codebook and factorization scheme to improve second order pooling. This paper presents a way to combine existing factorized second order representations with a codebook style hard assignment. Proposal for a novel bilinear representation based on a codebook model, and an efficient formulation in which codebook-based projections are factorized via shared projection to further reduce parameter size.