Small Temperature is All You Need for Differentiable Architecture Search
Jiuling Zhang Zhiming Ding
Differentiable architecture search (DARTS) yields highly efficient
gradient-based neural architecture search (NAS) by relaxing the discrete
operation selection to optimize continuous architecture parameters that maps
NAS from the discrete optimization to a continuous problem. DARTS then remaps
the relaxed supernet back to the discrete space by one-off post-search pruning
to obtain the final architecture (finalnet). Some emerging works argue that
this remap is inherently prone to mismatch the network between training and
evaluation which leads to performance discrepancy and even model collapse in
extreme cases. We propose to close the gap between the relaxed supernet in
training and the pruned finalnet in evaluation through utilizing small
temperature to sparsify the continuous distribution in the training phase. To
this end, we first formulate sparse-noisy softmax to get around gradient
saturation. We then propose an exponential temperature schedule to better
control the outbound distribution and elaborate an entropy-based adaptive
scheme to finally achieve the enhancement. We conduct extensive experiments to
verify the efficiency and efficacy of our method.