Weight-Entanglement Meets Gradient-Based Neural Architecture Search
Rhea Sanjay Sukthanker Arjun Krishnakumar Mahmoud Safari Frank Hutter
Weight sharing is a fundamental concept in neural architecture search (NAS),
enabling gradient-based methods to explore cell-based architecture spaces
significantly faster than traditional blackbox approaches. In parallel, weight
\emph{entanglement} has emerged as a technique for intricate parameter sharing
among architectures within macro-level search spaces. %However, the macro
structure of such spaces poses compatibility challenges for gradient-based NAS
methods. %As a result, blackbox optimization methods have been commonly
employed, particularly in conjunction with supernet training, to maintain
search efficiency. %Due to the inherent differences in the structure of these
search spaces, these Since weight-entanglement poses compatibility challenges
for gradient-based NAS methods, these two paradigms have largely developed
independently in parallel sub-communities. This paper aims to bridge the gap
between these sub-communities by proposing a novel scheme to adapt
gradient-based methods for weight-entangled spaces. This enables us to conduct
an in-depth comparative assessment and analysis of the performance of
gradient-based NAS in weight-entangled search spaces. Our findings reveal that
this integration of weight-entanglement and gradient-based NAS brings forth the
various benefits of gradient-based methods (enhanced performance, improved
supernet training properties and superior any-time performance), while
preserving the memory efficiency of weight-entangled spaces. The code for our
work is openly accessible
\href{https://anonymous.4open.science/r/TangleNAS-527C}{here}