AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search
Junghyup Lee Bumsub Ham
Training-free network architecture search (NAS) aims to discover
high-performing networks with zero-cost proxies, capturing network
characteristics related to the final performance. However, network rankings
estimated by previous training-free NAS methods have shown weak correlations
with the performance. To address this issue, we propose AZ-NAS, a novel
approach that leverages the ensemble of various zero-cost proxies to enhance
the correlation between a predicted ranking of networks and the ground truth
substantially in terms of the performance. To achieve this, we introduce four
novel zero-cost proxies that are complementary to each other, analyzing
distinct traits of architectures in the views of expressivity, progressivity,
trainability, and complexity. The proxy scores can be obtained simultaneously
within a single forward and backward pass, making an overall NAS process highly
efficient. In order to integrate the rankings predicted by our proxies
effectively, we introduce a non-linear ranking aggregation method that
highlights the networks highly-ranked consistently across all the proxies.
Experimental results conclusively demonstrate the efficacy and efficiency of
AZ-NAS, outperforming state-of-the-art methods on standard benchmarks, all
while maintaining a reasonable runtime cost.