How Much Is Hidden in the NAS Benchmarks? Few-Shot Adaptation of a NAS
  Predictor
Hrushikesh Loya Łukasz Dudziak Abhinav Mehrotra Royson Lee Javier Fernandez-Marques Nicholas D. Lane Hongkai Wen
Neural architecture search has proven to be a powerful approach to designing
and refining neural networks, often boosting their performance and efficiency
over manually-designed variations, but comes with computational overhead. While
there has been a considerable amount of research focused on lowering the cost
of NAS for mainstream tasks, such as image classification, a lot of those
improvements stem from the fact that those tasks are well-studied in the
broader context. Consequently, applicability of NAS to emerging and
under-represented domains is still associated with a relatively high cost
and/or uncertainty about the achievable gains. To address this issue, we turn
our focus towards the recent growth of publicly available NAS benchmarks in an
attempt to extract general NAS knowledge, transferable across different tasks
and search spaces. We borrow from the rich field of meta-learning for few-shot
adaptation and carefully study applicability of those methods to NAS, with a
special focus on the relationship between task-level correlation (domain shift)
and predictor transferability; which we deem critical for improving NAS on
diverse tasks. In our experiments, we use 6 NAS benchmarks in conjunction,
spanning in total 16 NAS settings -- our meta-learning approach not only shows
superior (or matching) performance in the cross-validation experiments but also
successful extrapolation to a new search space and tasks.