Neural Architecture Transfer 2: A Paradigm for Improving Efficiency in
  Multi-Objective Neural Architecture Search
Simone Sarti Eugenio Lomurno Matteo Matteucci
Deep learning is increasingly impacting various aspects of contemporary
society. Artificial neural networks have emerged as the dominant models for
solving an expanding range of tasks. The introduction of Neural Architecture
Search (NAS) techniques, which enable the automatic design of task-optimal
networks, has led to remarkable advances. However, the NAS process is typically
associated with long execution times and significant computational resource
requirements. Once-For-All (OFA) and its successor, Once-For-All-2 (OFAv2),
have been developed to mitigate these challenges. While maintaining exceptional
performance and eliminating the need for retraining, they aim to build a single
super-network model capable of directly extracting sub-networks satisfying
different constraints. Neural Architecture Transfer (NAT) was developed to
maximise the effectiveness of extracting sub-networks from a super-network. In
this paper, we present NATv2, an extension of NAT that improves multi-objective
search algorithms applied to dynamic super-network architectures. NATv2
achieves qualitative improvements in the extractable sub-networks by exploiting
the improved super-networks generated by OFAv2 and incorporating new policies
for initialisation, pre-processing and updating its networks archive. In
addition, a post-processing pipeline based on fine-tuning is introduced.
Experimental results show that NATv2 successfully improves NAT and is highly
recommended for investigating high-performance architectures with a minimal
number of parameters.