The process of designing neural architectures requires expert knowledge and extensive trial and error.
 While automated architecture search may simplify these requirements, the recurrent neural network (RNN) architectures generated by existing methods are limited in both flexibility and components.
 We propose a domain-specific language (DSL) for use in automated architecture search which can produce novel RNNs of arbitrary depth and width.
 The DSL is flexible enough to define standard architectures such as the Gated Recurrent Unit and Long Short Term Memory and allows the introduction of non-standard RNN components such as trigonometric curves and layer normalization.   Using two different candidate generation techniques, random search with a ranking function and reinforcement learning, 
we explore the novel architectures produced by the RNN DSL for language modeling and machine translation domains.
 The resulting architectures do not follow human intuition yet perform well on their targeted tasks, suggesting the space of usable RNN architectures is far larger than previously assumed. Developing novel neural network architectures is at the core of many recent AI advances BID28 BID14 BID35 . The process of architecture search and engineering is slow, costly, and laborious. Human experts, guided by intuition, explore an extensive space of potential architectures where even minor modifications can produce unexpected results. Ideally, an automated architecture search algorithm would find the optimal model architecture for a given task.Many explorations into the automation of machine learning have been made, including the optimization of hyperparameters BID3 BID24 and various methods of producing novel model architectures BID27 BID1 Zoph and Le, 2017) . For architecture search, ensuring these automated methods are able to produce results similar to humans usually requires traversing an impractically large search space, assuming high quality architectures exist in the search space at all. The choice of underlying operators composing an architecture is further typically constrained to a standard set across architectures even though recent work has found promising results in the use of non-standard operators BID31 .We propose a meta-learning strategy for flexible automated architecture search of recurrent neural networks (RNNs) which explicitly includes novel operators in the search. It consists of three stages, outlined in Figure 1 , for which we instantiate two versions.1. A candidate architecture generation function produces potential RNN architectures using a highly flexible DSL. The DSL enforces no constraints on the size or complexity of the generated tree and can be incrementally constructed using either a random policy or with an RL agent. 2. A ranking function processes each candidate architecture's DSL via a recursive neural network, predicting the architecture's performance. By unrolling the RNN representation, the ranking function can also model the interactions of a candidate architecture's hidden state over time. Figure 1: A generator produces candidate architectures by iteratively sampling the next node (either randomly or using an RL agent trained with REINFORCE). Full architectures are processed by a ranking function and the most promising candidates are evaluated. The results from running the model against a baseline experiment are then used to improve the generator and the ranking function.3. An evaluator, which takes the most promising candidate architectures, compiles their DSLs to executable code and trains each model on a specified task. The results of these evaluations form architecture-performance pairs that are then used to train the ranking function and RL generator. We introduced a flexible domain specific language for defining recurrent neural network architectures that can represent most human designed architectures. It is this flexibility that allowed our generators to come up with novel combinations in two tasks. These architectures used both core operators that are already used in current architectures as well as operators that are largely unstudied such as division or sine curves. The resulting architectures do not follow human intuition yet perform well on their targeted tasks, suggesting the space of usable RNN architectures is far larger than previously assumed. We also introduce a component-based concept for architecture search from which we instantiated two approaches: a ranking function driven search which allows for richer representations of complex RNN architectures that involve long term memory (c t ) nodes, and a Reinforcement Learning agent that internalizes knowledge about the search space to propose increasingly better architectures. As computing resources continue to grow, we see automated architecture generation as a promising avenue for future research.APPENDIX A: DOMAIN SPECIFIC LANGUAGE DISPLAYFORM0
We define a flexible DSL for RNN architecture generation that allows RNNs of varying size and complexity and propose a ranking function that represents RNNs as recursive neural networks, simulating their performance to decide on the most promising architectures. Introduces a new method to generate RNNs architectures using a domain-specific language for two types of generators (random and RL-based) together with a ranking function and evaluator. This paper casts the search of good RNN Cell architectures as a black-box optimization problem where examples are represented as an operator tree and scored based on learnt functions or generated by a RL agent. This paper investigates meta-learning strategy for automated architecture search in the context of RNN by using a DSL that specifies RNN recurrent operations.