Successful training of convolutional neural networks is often associated with suffi-
 ciently deep architectures composed of high amounts of features. These networks
 typically rely on a variety of regularization and pruning techniques to converge
 to less redundant states. We introduce a novel bottom-up approach to expand
 representations in fixed-depth architectures. These architectures start from just a
 single feature per layer and greedily increase width of individual layers to attain
 effective representational capacities needed for a specific task. While network
 growth can rely on a family of metrics, we propose a computationally efficient
 version based on feature time evolution and demonstrate its potency in determin-
 ing feature importance and a networks’ effective capacity. We demonstrate how
 automatically expanded architectures converge to similar topologies that benefit
 from lesser amount of parameters or improved accuracy and exhibit systematic
 correspondence in representational complexity with the specified task. In contrast
 to conventional design patterns with a typical monotonic increase in the amount of
 features with increased depth, we observe that CNNs perform better when there is
 more learnable parameters in intermediate, with falloffs to earlier and later layers. Estimating and consequently adequately setting representational capacity in deep neural networks for any given task has been a long standing challenge. Fundamental understanding still seems to be insufficient to rapidly decide on suitable network sizes and architecture topologies. While widely adopted convolutional neural networks (CNNs) such as proposed by BID17 ; BID27 ; BID12 ; Zagoruyko & Komodakis (2016) demonstrate high accuracies on a variety of problems, the memory footprint and computational complexity vary. An increasing amount of recent work is already providing valuable insights and proposing new methodology to address these points. For instance, the authors of BID2 propose a reinforcement learning based meta-learning approach to have an agent select potential CNN layers in a greedy, yet iterative fashion. Other suggested architecture selection algorithms draw their inspiration from evolutionary synthesis concepts BID25 BID22 . Although the former methods are capable of evolving architectures that rival those crafted by human design, it is currently only achievable at the cost of navigating large search spaces and hence excessive computation and time. As a trade-off in present deep neural network design processes it thus seems plausible to consider layer types or depth of a network to be selected by an experienced engineer based on prior knowledge and former research. A variety of techniques therefore focus on improving already well established architectures. Procedures ranging from distillation of one network's knowledge into another , compressing and encoding learned representations BID8 , pruning alongside potential re-training of networks BID7 BID26 BID10 , small capacity increases on trained networks in transfer-learning scenarios (WardeFarley et al., 2014) and the employment of different regularization terms during training BID11 BID15 BID23 BID0 , are just a fraction of recent efforts in pursuit of reducing representational complexity while attempting to retain accuracy. Underlying mechanisms rely on a multitude of criteria such as activation magnitudes BID26 and small weight values BID7 that are used as pruning metrics for either single neurons or complete feature maps, in addition to further combination with regularization and penalty terms.Common to these approaches is the necessity of training networks with large parameter quantities for maximum representational capacity to full convergence and the lack of early identification of insufficient capacity. In contrast, this work proposes a bottom-up approach with the following contributions:• We introduce a computationally efficient, intuitive metric to evaluate feature importance at any point of training a neural network. The measure is based on feature time evolution, specifically the normalized cross-correlation of each feature with its initialization state.• We propose a bottom-up greedy algorithm to automatically expand fixed-depth networks that start with one feature per layer until adequate representational capacity is reached. We base addition of features on our newly introduced metric due to its computationally efficient nature, while in principle a family of similarly constructed metrics is imaginable.• We revisit popular CNN architectures and compare them to automatically expanded networks. We show how our architectures systematically scale in terms of complexity of different datasets and either maintain their reference accuracy at reduced amount of parameters or achieve better results through increased network capacity.• We provide insights on how evolved network topologies differ from their reference counterparts where conventional design commonly increases the amount of features monotonically with increasing network depth. We observe that expanded architectures exhibit increased feature counts at early to intermediate layers and then proceed to decrease in complexity. In this work we have introduced a novel bottom-up algorithm to start neural network architectures with one feature per layer and widen them until a task depending suitable representational capacity is achieved. For the use in this framework we have presented one potential computationally efficient and intuitive metric to gauge feature importance. The proposed algorithm is capable of expanding architectures that provide either reduced amount of parameters or improved accuracies through higher amount of representations. This advantage seems to be gained through alternative network topologies with respect to commonly applied designs in current literature. Instead of increasing the amount of features monotonically with increasing depth of the network, we empirically observe that expanded neural network topologies have high amount of representations in early to intermediate layers.Future work could include a re-evaluation of plainly stacked deep architectures with new insights on network topologies and extended evaluation on different domain data. We have furthermore started to replace the currently present re-initialization step in the proposed expansion algorithm by keeping learned filters. In principle this approach looks promising but does need further systematic analysis of new feature initialization with respect to the already learned feature subset and accompanied investigation of orthogonality to avoid falling into local minima.
A bottom-up algorithm that expands CNNs starting with one feature per layer to architectures with sufficient representational capacity. Proposes to dynamically adjust the feature map depth of a fully convolutional neural network, formulating a measure of self-resemblance and boosting performance. Introduces a simple correlation-based metric to measure whether filters in neural networks are being used effectively, as a proxy for effective capacity. Aims to address the deep learning architecture search problem via incremental addition and removal of channels in intermediate layers of the network.