Auto-Prox: Training-Free Vision Transformer Architecture Search via
  Automatic Proxy Discovery
Zimian Wei Lujun Li Peijie Dong Zheng Hui Anggeng Li Menglong Lu Hengyue Pan Zhiliang Tian Dongsheng Li
The substantial success of Vision Transformer (ViT) in computer vision tasks
is largely attributed to the architecture design. This underscores the
necessity of efficient architecture search for designing better ViTs
automatically. As training-based architecture search methods are
computationally intensive, there is a growing interest in training-free methods
that use zero-cost proxies to score ViTs. However, existing training-free
approaches require expert knowledge to manually design specific zero-cost
proxies. Moreover, these zero-cost proxies exhibit limitations to generalize
across diverse domains. In this paper, we introduce Auto-Prox, an automatic
proxy discovery framework, to address the problem. First, we build the
ViT-Bench-101, which involves different ViT candidates and their actual
performance on multiple datasets. Utilizing ViT-Bench-101, we can evaluate
zero-cost proxies based on their score-accuracy correlation. Then, we represent
zero-cost proxies with computation graphs and organize the zero-cost proxy
search space with ViT statistics and primitive operations. To discover generic
zero-cost proxies, we propose a joint correlation metric to evolve and mutate
different zero-cost proxy candidates. We introduce an elitism-preserve strategy
for search efficiency to achieve a better trade-off between exploitation and
exploration. Based on the discovered zero-cost proxy, we conduct a ViT
architecture search in a training-free manner. Extensive experiments
demonstrate that our method generalizes well to different datasets and achieves
state-of-the-art results both in ranking correlation and final accuracy. Codes
can be found at https://github.com/lilujunai/Auto-Prox-AAAI24.