We investigate in this paper the architecture of deep convolutional networks. Building on existing state of the art models, we propose a reconfiguration of the model parameters into several parallel branches at the global network level, with each branch being a standalone CNN. We show that this arrangement is an efficient way to significantly reduce the number of parameters while at the same time improving the performance. The use of branches brings an additional form of regularization. In addition to splitting the parameters into parallel branches, we propose a tighter coupling of these branches by averaging their log-probabilities. The tighter coupling favours the learning of better representations, even at the level of the individual branches, as compared to when each branch is trained independently. We refer to this branched architecture as "coupled ensembles". The approach is very generic and can be applied with almost any neural network architecture. With coupled ensembles of DenseNet-BC and parameter budget of 25M, we obtain error rates of 2.92%, 15.68% and 1.50% respectively on CIFAR-10, CIFAR-100 and SVHN tasks. For the same parameter budget, DenseNet-BC has an error rate of 3.46%, 17.18%, and 1.8% respectively.   With ensembles of coupled ensembles, of DenseNet-BC networks, with 50M total parameters, we obtain error rates of 2.72%, 15.13% and 1.42% respectively on these tasks. The design of early convolutional architectures (CNN) involved choices of hyper-parameters such as: filter size, number of filters at each layer, and padding BID15 BID14 . Since the introduction of the VGGNet BID20 ) the design has moved towards following a template: fixed filter size of 3 × 3 and N features maps, down-sample to half the input resolution only by the use of either maxpool or strided convolutions BID21 , and double the number the computed feature maps following each down-sampling operation. This philosophy is used by state of the art models like ResNet BID8 and DenseNet BID11 . The last two architectures extended the template to include the use of "skip-connections" between non-contiguous layers.Our work extends this template by adding another element, which we refer to as "coupled ensembling". In this set-up, the network is decomposed into several branches, each branch being functionally similar to a complete CNN. The proposed template achieves performance comparable to state of the art models with a significantly lower parameter count.In this paper, we make the following contributions: (i) we show that given a parameter budget, it is better to have the parameters split among branches rather than having a single branch (which is the case for all current networks), (ii) we compare different ways to combine the activations of the parallel branches and find that it is best to take an arithmetic mean of the individual logprobabilities (iii) combining these elements, we significantly match and improve the performance of convolutional networks on CIFAR and SVHN datasets, with a heavily reduced parameter count. (iv) Further ensembling of coupled ensembles lead to more improvement. This paper is organised as follows: in section 2, we discuss related work; in section 3, we introduce the concept of coupled ensembles and the motivation behind the idea; in section 4, we evaluate the proposed approach and compare it with the state of the art; and we conclude and discuss future work in section 5. The proposed approach consists in replacing a single deep convolutional network by a number of "element blocks" which resemble standalone CNN models. The intermediate score vectors produced by each of the elements blocks are coupled via a "fuse layer". At training time, this is done by taking an arithmetic average of their log-probabilities for the targets. At test time the score vectors are averaged following the output from each score vector. Both of these aspects leads to a significant performance improvement over a single branch configuration. This improvement comes at the cost of a small increase in the training and prediction times. The proposed approach leads to the best performance for a given parameter budget as can be seen in tables 3 and 4, and in figure 2. Additionally, the individual "element block" performance is better as compared to when they are trained independently.The increase in training and prediction times is mostly due to the sequential processing of branches during the forward and backward passes. The smaller size of the branches makes the data parallelism on GPUs less efficient. This effect is not as pronounced for larger models. This could be solved in two ways. First, as there is no data dependency between the branches (before the averaging layer) it is possible to extend the data parallelism to the branches, restoring the initial level of parallelism. This can be done by through a parallel implementation of multiple 2D convolutions at the same time. Second or alternatively, when multiple GPUs are used, it is possible to spread the branches over the GPUs.Preliminary experiments on ImageNet (Russakovsky et al., 2015) show that coupled ensembles have a lower error for the same parameter budget as compared to single branch models. We will expand on these experiments in the future.
We show that splitting a neural network into parallel branches improves performance and that proper coupling of the branches improves performance even further. The work proposes a reconfiguration of the existing state-of-the-art CNN model using a new branching architecture, with better performance. This paper shows parameter-saving benefits of coupled ensembling. Presents a deep network architecture which processes data using multiple parallel branches and combines the posterior from these branches to compute the final scores.