The design of small molecules with bespoke properties is of central importance to drug discovery.   However significant challenges yet remain for computational methods, despite recent advances such as deep recurrent networks and reinforcement learning strategies for sequence generation, and it can be difficult to compare results across different works.   This work proposes 19 benchmarks selected by subject experts, expands smaller datasets previously used to approximately 1.1 million training molecules, and explores how to apply new reinforcement learning techniques effectively for molecular design.   The benchmarks here, built as OpenAI Gym environments, will be open-sourced to encourage innovation in molecular design algorithms and to enable usage by those without a background in chemistry.   Finally, this work explores recent development in reinforcement-learning methods with excellent sample complexity (the A2C and PPO algorithms) and investigates their behavior in molecular generation, demonstrating significant performance gains compared to standard reinforcement learning techniques. Novel drugs are developed using design -make -test cycles: molecules are designed, synthesized in the laboratory, and then tested for their biological effect. The insights gained from these tests then inform the design for the next iteration. The objective of de novo design methodologies is to perform this cycle with computational methods BID4 BID29 . The test phase was the first to be automated, using the broad categorization of machine learning models known as quantitative structure-activity/property relationships (QSAR/QSPR) to predict the activity of a molecule against a certain biological target, or physicochemical properties. To make virtual molecules, symbolic approaches based on graph rewriting have been used, which are domain-specific and rely on extensive hand-engineering by experts. To optimize the properties of a molecule, for example its activity against a biological target (design), global optimization approaches such as evolutionary algorithms or ant colony optimization have been used BID4 BID29 . Symbolic approaches have been highlighted as either generating unrealistic molecules that would be difficult to synthesize, or for being too conservative, and therefore not sufficiently exploring the space of tractable molecules BID29 BID2 .Recently , generative models have been proposed to learn the distribution of real druglike molecules from data, and then to generate chemical structures that are appropriate for the application domain BID36 . Interestingly , the generation of molecules is related to natural language generation (NLG) . Two classic problems of NLG -preserving coherent long-range dependencies, and syntactic and semantic correctness -directly map to molecules. Current investigations draw heavily from tools developed for language tasks, including variational autoencoders (VAE) BID12 BID18 , recurrent neural network (RNN) models BID32 BID17 BID24 , generative adversarial networks (GAN) BID15 and Monte Carlo Tree Search (MCTS) BID38 .This work seeks to consolidate the growing body of recurrent models for molecular design that employ reinforcement learning. Here, we suggest a set of 19 benchmarks of relevance to de novo design. Furthermore, an implementation of these benchmarks as an OpenAI Gym is provided to the community to spur further innovation. Finally, we demonstrate state-of-the-art performance using new techniques drawing from recent advances in reinforcement learning.Under review as a conference paper at ICLR 2018 In this work, we proposed a large, standardized dataset and a set of 19 benchmarks to evaluate models for molecule generation and design. Several RL strategies were investigated on these benchmarks. Here, the results suggest that the Hillclimb-MLE model is a surprisingly robust technique with large datasets and deep models, outperforming PPO given sufficient compute times and sample evaluations. In the space of constrained compute and sample evaluations, PPO was shown to be an effective learning algorithm, converging an order of magnitude before other reinforcement-learning algorithms for molecule generation.Nevertheless, there is still tremendous need for more efficient and effective models for molecular design, which could have a profound impact on molecular design -including drug, materials and agrochemicals discovery -and thus immediately on human well-being. With the present, easily usable benchmark, we hope to inspire the machine learning community to pick up this challenge.
We investigate a variety of RL algorithms for molecular generation and define new benchmarks (to be released as an OpenAI Gym), finding PPO and a hill-climbing MLE algorithm work best. Considers model evaluation for molecule generation by proposing 19 benchmarks, expanding small data sets to a large, standardized dataset, and exploring how to apply RL techniques for molecular design. This paper shows that the most sophisticated RL methods are less effective than the simple hill-climbing technique, with PPO as the exception, when modeling and synthesizing molecules.