ParZC: Parametric Zero-Cost Proxies for Efficient NAS
Peijie Dong Lujun Li Xinglin Pan Zimian Wei Xiang Liu Qiang Wang Xiaowen Chu
Recent advancements in Zero-shot Neural Architecture Search (NAS) highlight
the efficacy of zero-cost proxies in various NAS benchmarks. Several studies
propose the automated design of zero-cost proxies to achieve SOTA performance
but require tedious searching progress. Furthermore, we identify a critical
issue with current zero-cost proxies: they aggregate node-wise zero-cost
statistics without considering the fact that not all nodes in a neural network
equally impact performance estimation. Our observations reveal that node-wise
zero-cost statistics significantly vary in their contributions to performance,
with each node exhibiting a degree of uncertainty. Based on this insight, we
introduce a novel method called Parametric Zero-Cost Proxies (ParZC) framework
to enhance the adaptability of zero-cost proxies through parameterization. To
address the node indiscrimination, we propose a Mixer Architecture with
Bayesian Network (MABN) to explore the node-wise zero-cost statistics and
estimate node-specific uncertainty. Moreover, we propose DiffKendall as a loss
function to directly optimize Kendall's Tau coefficient in a differentiable
manner so that our ParZC can better handle the discrepancies in ranking
architectures. Comprehensive experiments on NAS-Bench-101, 201, and NDS
demonstrate the superiority of our proposed ParZC compared to existing
zero-shot NAS methods. Additionally, we demonstrate the versatility and
adaptability of ParZC by transferring it to the Vision Transformer search
space.