ICDARTS: Improving the Stability and Performance of Cyclic DARTS
Emily Herron Derek Rose Steven Young
This work introduces improvements to the stability and generalizability of
Cyclic DARTS (CDARTS). CDARTS is a Differentiable Architecture Search
(DARTS)-based approach to neural architecture search (NAS) that uses a cyclic
feedback mechanism to train search and evaluation networks concurrently. This
training protocol aims to optimize the search process by enforcing that the
search and evaluation networks produce similar outputs. However, CDARTS
introduces a loss function for the evaluation network that is dependent on the
search network. The dissimilarity between the loss functions used by the
evaluation networks during the search and retraining phases results in a
search-phase evaluation network that is a sub-optimal proxy for the final
evaluation network that is utilized during retraining. We present ICDARTS, a
revised approach that eliminates the dependency of the evaluation network
weights upon those of the search network, along with a modified process for
discretizing the search network's \textit{zero} operations that allows these
operations to be retained in the final evaluation networks. We pair the results
of these changes with ablation studies on ICDARTS' algorithm and network
template. Finally, we explore methods for expanding the search space of ICDARTS
by expanding its operation set and exploring alternate methods for discretizing
its continuous search cells. These experiments resulted in networks with
improved generalizability and the implementation of a novel method for
incorporating a dynamic search space into ICDARTS.