Robustifying DARTS by Eliminating Information Bypass Leakage via
  Explicit Sparse Regularization
Jiuling Zhang Zhiming Ding
Differentiable architecture search (DARTS) is a promising end to end NAS
method which directly optimizes the architecture parameters through general
gradient descent. However, DARTS is brittle to the catastrophic failure
incurred by the skip connection in the search space. Recent studies also cast
doubt on the basic underlying hypotheses of DARTS which are argued to be
inherently prone to the performance discrepancy between the continuous-relaxed
supernet in the training phase and the discretized finalnet in the evaluation
phase. We figure out that the robustness problem and the skepticism can both be
explained by the information bypass leakage during the training of the
supernet. This naturally highlights the vital role of the sparsity of
architecture parameters in the training phase which has not been well developed
in the past. We thus propose a novel sparse-regularized approximation and an
efficient mixed-sparsity training scheme to robustify DARTS by eliminating the
information bypass leakage. We subsequently conduct extensive experiments on
multiple search spaces to demonstrate the effectiveness of our method.