Recent work has shown that performing inference with fast, very-low-bitwidth
 (e.g., 1 to 2 bits) representations of values in models can yield surprisingly accurate
 results. However, although 2-bit approximated networks have been shown to
 be quite accurate, 1 bit approximations, which are twice as fast, have restrictively
 low accuracy. We propose a method to train models whose weights are a mixture
 of bitwidths, that allows us to more finely tune the accuracy/speed trade-off. We
 present the “middle-out” criterion for determining the bitwidth for each value, and
 show how to integrate it into training models with a desired mixture of bitwidths.
 We evaluate several architectures and binarization techniques on the ImageNet
 dataset. We show that our heterogeneous bitwidth approximation achieves superlinear
 scaling of accuracy with bitwidth. Using an average of only 1.4 bits, we are
 able to outperform state-of-the-art 2-bit architectures. With Convolutional Neural Nets (CNNs) now outperforming humans in vision classification tasks BID11 , it is clear that CNNs will be a mainstay of AI applications. However, CNNs are known to be computationally demanding, and are most comfortably run on GPUs. For execution in mobile and embedded settings, or when a given CNN is evaluated many times, using a GPU may be too costly. The search for inexpensive variants of CNNs has yielded techniques such as hashing BID0 , vector quantization BID4 , and pruning BID5 . One particularly promising track is binarization BID1 , which replaces 32-bit floating point values with single bits, either +1 or -1, and (optionally) replaces floating point multiplies with packed bitwise popcount-xnors . Binarization can reduce the size of models by up to 32×, and reduce the number of operations executed by up to 64×.Binarized CNNs are faster and smaller, but also less accurate. Much research has therefore focused on reducing the accuracy gap between binary models and their floating point counterparts. The typical approach is to add bits to the activations and weights of a network, giving a better approximation of the true values. However, the cost of extra bits is quite high. Using n bits to approximate just the weights increases the computation and memory required by a factor of n compared to 1-bit binarization. Further using n bits to approximate activations as well requires n 2 times the resources as one bit. There is thus a strong motivation to use as few bits as possible while still achieving acceptable accuracy. However, today's binary approximations are locked to use the same number of bits for all approximated values, and the gap in accuracy between bits can be substantial. For example, recent work concludes 1-bit accuracy is unsatisfactory while 2-bit accuracy is quite high BID12 (also see TAB0 ).In order to bridge the gap between integer bits, we introduce Heterogeneous Bitwidth Neural Networks (HBNNs), which use a mix of integer bitwidths to allow values to have effectively (i.e., on average) fractional bitwidths. The freedom to select from multiple bitwidths allows HBNNs to approximate each value better than fixed-bitwidth schemes, giving them disproportionate accuracy gains for the number of effective bits used. For instance, Alexnet trained with an average of 1.4 bits has comparable (actually, slightly higher) accuracy to training with a fixed two bits TAB0 .Our main contributions are:(1) We propose HBNNs as a way to break the integer-bitwidth barrier in binarized networks.(2) We study several techniques for distributing the bitwidths in a HBNN, and introduce the middle-out bitwidth selection algorithm, which uses the full representational power of heterogeneous bitwidths to learn good bitwidth distributions. (3) We perform a comprehensive study of heterogeneous binarization on the ImageNet dataset using an AlexNet architecture. We evaluate many fractional bitwidths and compare to state of the art results. HBNNs typically yield the smallest and fastest networks at each accuracy. Further, we show that it is usually possible to equal, or improve upon, 2-bitbinarized networks with an average of 1.4 bits. (4) We show that heterogeneous binarization is applicable to MobileNet BID6 , demonstrating that its benefits apply even to modern, optimized architectures. In this paper, we present Heterogeneous Bitwidth Neural Networks (HBNNs), a new type of binary network that is not restricted to integer bitwidths. Allowing effectively fractional bitwidths in networks gives a vastly improved ability to tune the trade-offs between accuracy, compression, and speed that come with binarization. We show a simple method of distributing bits across a tensor lead to a linear relationship between accuracy and number of bits, but using a more informed method allows higher accuracy with fewer bits. We introduce middle-out bit selection as the top performing technique for determining where to place bits in a heterogeneous bitwidth tensor and find that Middle-Out enables a heterogeneous representation to be more powerful than a homogeneous one. On the ImageNet dataset with AlexNet and MobileNet models, we perform extensive experiments to validate the effectiveness of HBNNs compared to the state of the art and full precision accuracy. The results of these experiments are highly compelling, with HBNNs matching or outperforming competing binarization techniques while using fewer average bits. The use of HBNNs enables applications which require higher compression and speeds offered by a low bitwidth but also need the accuracy of a high bitwidth. As future work, we will investigate modifying the bit selection method to make heterogeneous bit tensors more amenable for CPU computation as well as develop a HBNN FPGA implementation which can showcase both the speed and accuracy benefits of heterogeneous binarization.
We introduce fractional bitwidth approximation and show it has significant advantages. Suggests a method for varying the degree of quantization in a neural network during the forward propagation phase Maintaining the accuracy of 2bits netword while using less than 2bits weights