ShiftNAS: Improving One-shot NAS via Probability Shift
Mingyang Zhang Xinyi Yu Haodong Zhao Linlin Ou
One-shot Neural architecture search (One-shot NAS) has been proposed as a
time-efficient approach to obtain optimal subnet architectures and weights
under different complexity cases by training only once. However, the subnet
performance obtained by weight sharing is often inferior to the performance
achieved by retraining. In this paper, we investigate the performance gap and
attribute it to the use of uniform sampling, which is a common approach in
supernet training. Uniform sampling concentrates training resources on subnets
with intermediate computational resources, which are sampled with high
probability. However, subnets with different complexity regions require
different optimal training strategies for optimal performance. To address the
problem of uniform sampling, we propose ShiftNAS, a method that can adjust the
sampling probability based on the complexity of subnets. We achieve this by
evaluating the performance variation of subnets with different complexity and
designing an architecture generator that can accurately and efficiently provide
subnets with the desired complexity. Both the sampling probability and the
architecture generator can be trained end-to-end in a gradient-based manner.
With ShiftNAS, we can directly obtain the optimal model architecture and
parameters for a given computational complexity. We evaluate our approach on
multiple visual network models, including convolutional neural networks (CNNs)
and vision transformers (ViTs), and demonstrate that ShiftNAS is
model-agnostic. Experimental results on ImageNet show that ShiftNAS can improve
the performance of one-shot NAS without additional consumption. Source codes
are available at https://github.com/bestfleer/ShiftNAS.