Multi-Objective Evolutionary Neural Architecture Search for Recurrent
  Neural Networks
Reinhard Booysen Anna Sergeevna Bosman
Artificial neural network (NN) architecture design is a nontrivial and
time-consuming task that often requires a high level of human expertise. Neural
architecture search (NAS) serves to automate the design of NN architectures and
has proven to be successful in automatically finding NN architectures that
outperform those manually designed by human experts. NN architecture
performance can be quantified based on multiple objectives, which include model
accuracy and some NN architecture complexity objectives, among others. The
majority of modern NAS methods that consider multiple objectives for NN
architecture performance evaluation are concerned with automated feed forward
NN architecture design, which leaves multi-objective automated recurrent neural
network (RNN) architecture design unexplored. RNNs are important for modeling
sequential datasets, and prominent within the natural language processing
domain. It is often the case in real world implementations of machine learning
and NNs that a reasonable trade-off is accepted for marginally reduced model
accuracy in favour of lower computational resources demanded by the model. This
paper proposes a multi-objective evolutionary algorithm-based RNN architecture
search method. The proposed method relies on approximate network morphisms for
RNN architecture complexity optimisation during evolution. The results show
that the proposed method is capable of finding novel RNN architectures with
comparable performance to state-of-the-art manually designed RNN architectures,
but with reduced computational demand.