Archtree: on-the-fly tree-structured exploration for latency-aware
  pruning of deep neural networks
Rémi Ouazan Reboul Edouard Yvinec Arnaud Dapogny Kevin Bailly
Deep neural networks (DNNs) have become ubiquitous in addressing a number of
problems, particularly in computer vision. However, DNN inference is
computationally intensive, which can be prohibitive e.g. when considering edge
devices. To solve this problem, a popular solution is DNN pruning, and more so
structured pruning, where coherent computational blocks (e.g. channels for
convolutional networks) are removed: as an exhaustive search of the space of
pruned sub-models is intractable in practice, channels are typically removed
iteratively based on an importance estimation heuristic. Recently, promising
latency-aware pruning methods were proposed, where channels are removed until
the network reaches a target budget of wall-clock latency pre-emptively
estimated on specific hardware. In this paper, we present Archtree, a novel
method for latency-driven structured pruning of DNNs. Archtree explores
multiple candidate pruned sub-models in parallel in a tree-like fashion,
allowing for a better exploration of the search space. Furthermore, it involves
on-the-fly latency estimation on the target hardware, accounting for closer
latencies as compared to the specified budget. Empirical results on several DNN
architectures and target hardware show that Archtree better preserves the
original model accuracy while better fitting the latency budget as compared to
existing state-of-the-art methods.