{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T16:28:49Z","timestamp":1776184129613,"version":"3.50.1"},"reference-count":110,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,10,3]],"date-time":"2022-10-03T00:00:00Z","timestamp":1664755200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,10,3]],"date-time":"2022-10-03T00:00:00Z","timestamp":1664755200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Cheminform"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    A plethora of AI-based techniques now exists to conduct de novo molecule generation that can devise molecules conditioned towards a particular endpoint in the context of drug design. One popular approach is using reinforcement learning to update a recurrent neural network or language-based de novo molecule generator. However, reinforcement learning can be inefficient, sometimes requiring up to 10\n                    <jats:sup>5<\/jats:sup>\n                    molecules to be sampled to optimize more complex objectives, which poses a limitation when using computationally expensive scoring functions like docking or computer-aided synthesis planning models. In this work, we propose a reinforcement learning strategy called Augmented Hill-Climb based on a simple, hypothesis-driven hybrid between REINVENT and Hill-Climb that improves sample-efficiency by addressing the limitations of both currently used strategies. We compare its ability to optimize several docking tasks with REINVENT and benchmark this strategy against other commonly used reinforcement learning strategies including REINFORCE, REINVENT (version 1 and 2), Hill-Climb and best agent reminder. We find that optimization ability is improved\u2009~\u20091.5-fold and sample-efficiency is improved\u2009~\u200945-fold compared to REINVENT while still delivering appealing chemistry as output. Diversity filters were used, and their parameters were tuned to overcome observed failure modes that take advantage of certain diversity filter configurations. We find that Augmented Hill-Climb outperforms the other reinforcement learning strategies used on six tasks, especially in the early stages of training or for more difficult objectives. Lastly, we show improved performance not only on recurrent neural networks but also on a reinforcement learning stabilized transformer architecture. Overall, we show that Augmented\u00a0Hill-Climb improves sample-efficiency for language-based de novo molecule generation conditioning via reinforcement learning, compared to the current state-of-the-art. This makes more computationally expensive scoring functions, such as docking, more accessible on a relevant timescale.\n                  <\/jats:p>","DOI":"10.1186\/s13321-022-00646-z","type":"journal-article","created":{"date-parts":[[2022,10,3]],"date-time":"2022-10-03T11:02:37Z","timestamp":1664794957000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":31,"title":["Augmented Hill-Climb increases reinforcement learning efficiency for language-based de novo molecule generation"],"prefix":"10.1186","volume":"14","author":[{"given":"Morgan","family":"Thomas","sequence":"first","affiliation":[]},{"given":"Noel M.","family":"O\u2019Boyle","sequence":"additional","affiliation":[]},{"given":"Andreas","family":"Bender","sequence":"additional","affiliation":[]},{"given":"Chris","family":"de Graaf","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,10,3]]},"reference":[{"key":"646_CR1","doi-asserted-by":"publisher","first-page":"1079","DOI":"10.1021\/ci034290p","volume":"44","author":"N Brown","year":"2004","unstructured":"Brown N, McKay B, Gilardoni F, Gasteiger J (2004) A graph-based genetic algorithm and its application to the multiobjective evolution of median molecules. J Chem Inf Comput Sci 44:1079\u20131087","journal-title":"J Chem Inf Comput Sci"},{"key":"646_CR2","doi-asserted-by":"publisher","first-page":"3567","DOI":"10.1039\/C8SC05372C","volume":"10","author":"JH Jensen","year":"2019","unstructured":"Jensen JH (2019) A graph-based genetic algorithm and generative model\/Monte Carlo tree search for the exploration of chemical space. Chem Sci 10:3567\u20133572","journal-title":"Chem Sci"},{"key":"646_CR3","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1021\/acscentsci.7b00512","volume":"4","author":"MHS Segler","year":"2018","unstructured":"Segler MHS, Kogej T, Tyrchan C, Waller MP (2018) Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS Cent Sci 4:120\u2013131","journal-title":"ACS Cent Sci"},{"key":"646_CR4","doi-asserted-by":"publisher","first-page":"48","DOI":"10.1186\/s13321-017-0235-x","volume":"9","author":"M Olivecrona","year":"2017","unstructured":"Olivecrona M, Blaschke T, Engkvist O, Chen H (2017) Molecular de-novo design through deep reinforcement learning. J Cheminform 9:48","journal-title":"J Cheminform"},{"issue":"7","key":"646_CR5","doi-asserted-by":"publisher","first-page":"eaap7885","DOI":"10.1126\/sciadv.aap7885","volume":"4","author":"M Popova","year":"2018","unstructured":"Popova M, Isayev O, Tropsha A (2018) Deep reinforcement learning for de novo drug design. Sci Adv 4(7):eaap7885","journal-title":"Sci Adv"},{"issue":"1","key":"646_CR6","doi-asserted-by":"publisher","first-page":"972","DOI":"10.1080\/14686996.2017.1401424","volume":"18","author":"X Yang","year":"2017","unstructured":"Yang X, Zhang J, Yoshizoe K et al (2017) ChemTS: an efficient python library for de novo molecular generation. Sci Technol Adv Mater\u00a018(1):972\u2013976","journal-title":"Sci Technol Adv Mater"},{"key":"646_CR7","doi-asserted-by":"publisher","first-page":"171","DOI":"10.1038\/s42256-020-0160-y","volume":"2","author":"M Moret","year":"2020","unstructured":"Moret M, Friedrich L, Grisoni F et al (2020) Generative molecular design in low data regimes. Nat Mach Intell 2:171\u2013180","journal-title":"Nat Mach Intell"},{"issue":"4","key":"646_CR8","doi-asserted-by":"publisher","first-page":"875","DOI":"10.1021\/acs.jcim.6b00754","volume":"57","author":"W Yuan","year":"2017","unstructured":"Yuan W, Jiang D, Nambiar DK et al (2017) Chemical space mimicry for drug discovery. J Chem Inf Model 57(4):875\u2013882","journal-title":"J Chem Inf Model"},{"key":"646_CR9","doi-asserted-by":"crossref","unstructured":"He J, You H, Sandstr\u00f6m E et al (2021) Molecular optimization by capturing chemist\u2019s intuition using deep neural networks. J Cheminform 13(26)","DOI":"10.1186\/s13321-021-00497-0"},{"issue":"3","key":"646_CR10","doi-asserted-by":"publisher","first-page":"914","DOI":"10.1038\/s42256-021-00403-1","volume":"310","author":"J Wang","year":"2021","unstructured":"Wang J, Hsieh C-Y, Wang M et al (2021) Multi-constraint molecular generation based on conditional transformer, knowledge distillation and reinforcement learning. Nat Mach Intell 310(3):914\u2013922","journal-title":"Nat Mach Intell"},{"issue":"9","key":"646_CR11","doi-asserted-by":"publisher","first-page":"2064","DOI":"10.1021\/acs.jcim.1c00600","volume":"62","author":"V Bagal","year":"2021","unstructured":"Bagal V, Aggarwal R, Vinod PK, Priyakumar UD (2021) MolGPT: molecular generation using a transformer-decoder model. J Chem Inf Model\u00a062(9):2064\u20132076","journal-title":"J Chem Inf Model"},{"issue":"2","key":"646_CR12","doi-asserted-by":"publisher","first-page":"268","DOI":"10.1021\/acscentsci.7b00572","volume":"4","author":"R G\u00f3mez-Bombarelli","year":"2018","unstructured":"G\u00f3mez-Bombarelli R, Wei JN, Duvenaud D et al (2018) Automatic chemical design using a data-driven continuous representation of molecules. ACS Cent Sci 4(2):268\u2013276","journal-title":"ACS Cent Sci"},{"key":"646_CR13","unstructured":"Jin W, Barzilay R, Jaakkola T (2018) Junction tree variational autoencoder for molecular graph generation. Proceedings of the 35th International Conference on Machine Learning, PMLR 80:2323\u20132332"},{"key":"646_CR14","unstructured":"Kajino H (2019) Molecular Hypergraph Grammar with its Application to Molecular Optimization. Proceedings of the 36th International Conference on Machine Learning, PMLR 97:3183\u20133191"},{"key":"646_CR15","doi-asserted-by":"publisher","first-page":"8016","DOI":"10.1039\/C9SC01928F","volume":"10","author":"R Winter","year":"2019","unstructured":"Winter R, Montanari F, Steffen A et al (2019) Efficient multi-objective molecular optimization in a continuous latent space. Chem Sci 10:8016\u20138024","journal-title":"Chem Sci"},{"key":"646_CR16","unstructured":"De Cao N, Kipf T (2018) MolGAN: An implicit generative model for small molecular graphs. arXiv preprint arXiv:1805.11973"},{"key":"646_CR17","unstructured":"Guimaraes GL, Sanchez-Lengeling B, Outeiral C, et al (2017) Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Models. arXiv preprint arXiv:1705.10843"},{"key":"646_CR18","doi-asserted-by":"crossref","unstructured":"Blanchard AE, Stanley C, Bhowmik D (2021) Using GANs with adaptive training data to search for new molecules. J Cheminform 13(14)","DOI":"10.1186\/s13321-021-00494-3"},{"key":"646_CR19","doi-asserted-by":"publisher","first-page":"025023","DOI":"10.1088\/2632-2153\/abcf91","volume":"2","author":"R Mercado","year":"2021","unstructured":"Mercado R, Rastemo T, Lindelof E et al (2021) Graph networks for molecular design. Mach Learn Sci Technol 2:025023","journal-title":"Mach Learn Sci Technol"},{"key":"646_CR20","doi-asserted-by":"publisher","unstructured":"Atance SR, Diez JV, Engkvist O, et al (2021) De novo drug design using reinforcement learning with graph-based deep generative models. ChemRxiv. https:\/\/doi.org\/10.26434\/chemrxiv-2021-9w3tc","DOI":"10.26434\/chemrxiv-2021-9w3tc"},{"issue":"1","key":"646_CR21","doi-asserted-by":"publisher","first-page":"10752","DOI":"10.1038\/s41598-019-47148-x","volume":"9","author":"Z Zhou","year":"2019","unstructured":"Zhou Z, Kearnes S, Li L et al (2019) Optimization of molecules via deep reinforcement learning. Sci Rep 9(1):10752","journal-title":"Sci Rep"},{"issue":"3","key":"646_CR22","doi-asserted-by":"publisher","first-page":"1096","DOI":"10.1021\/acs.jcim.8b00839","volume":"59","author":"N Brown","year":"2019","unstructured":"Brown N, Fiscato M, Segler MHS, Vaucher AC (2019) GuacaMol: benchmarking models for de novo molecular design. J Chem Inf Model 59(3):1096\u20131108","journal-title":"J Chem Inf Model"},{"key":"646_CR23","doi-asserted-by":"publisher","first-page":"565644","DOI":"10.3389\/fphar.2020.565644","volume":"11","author":"D Polykovskiy","year":"2020","unstructured":"Polykovskiy D, Zhebrak A, Sanchez-Lengeling B et al (2020) Molecular sets (MOSES): a benchmarking platform for molecular generation models. Front Pharmacol 11:565644","journal-title":"Front Pharmacol"},{"key":"646_CR24","unstructured":"Popova M, Shvets M, Oliva J, Isayev O (2019) MolecularRNN: Generating realistic molecular graphs with optimized properties. arXiv preprint arXiv:1905.13372"},{"key":"646_CR25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/978-1-0716-1787-8_1","volume":"2390","author":"M Thomas","year":"2022","unstructured":"Thomas M, Boardman A, Garcia-Ortegon M et al (2022) Applications of artificial intelligence in drug design: opportunities and challenges. Methods Mol Biol 2390:1\u201359","journal-title":"Methods Mol Biol"},{"key":"646_CR26","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1021\/ci00057a005","volume":"28","author":"D Weininger","year":"1988","unstructured":"Weininger D (1988) SMILES, a chemical language and information system: 1: introduction to methodology and encoding rules. J Chem Inf Comput Sci 28:31\u201336","journal-title":"J Chem Inf Comput Sci"},{"key":"646_CR27","unstructured":"Jozefowicz R, Vinyals O, Schuster M, et al (2016) Exploring the Limits of Language Modeling. arXiv preprint arXiv:1602.02410"},{"key":"646_CR28","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1007\/978-3-540-27835-1_10","volume":"3141","author":"A Graves","year":"2004","unstructured":"Graves A, Eck D, Beringer N, Schmidhuber J (2004) Biologically plausible speech recognition with LSTM neural nets. Lect Notes Comput Sci 3141:127\u2013136","journal-title":"Lect Notes Comput Sci"},{"key":"646_CR29","doi-asserted-by":"crossref","unstructured":"Liu X, Ye K, van Vlijmen HWT et al (2021) DrugEx v2: de novo design of drug molecules by Pareto-based multi-objective reinforcement learning in polypharmacology. J Cheminform 13(8)","DOI":"10.1186\/s13321-021-00561-9"},{"issue":"9","key":"646_CR30","doi-asserted-by":"publisher","first-page":"1038","DOI":"10.1038\/s41587-019-0224-x","volume":"37","author":"A Zhavoronkov","year":"2019","unstructured":"Zhavoronkov A, Ivanenkov YA, Aliper A et al (2019) Deep learning enables rapid identification of potent DDR1 kinase inhibitors. Nat Biotechnol 37(9):1038\u20131040","journal-title":"Nat Biotechnol"},{"issue":"9","key":"646_CR31","doi-asserted-by":"publisher","first-page":"2093","DOI":"10.1021\/acs.jcim.1c00777","volume":"62","author":"R Mercado","year":"2021","unstructured":"Mercado R, Bjerrum EJ, Engkvist O (2022) Exploring graph traversal algorithms in graph-based molecular generation. J Chem Inf Model 62(9):2093\u20132100","journal-title":"J Chem Inf Model"},{"issue":"35","key":"646_CR32","doi-asserted-by":"publisher","first-page":"19477","DOI":"10.1002\/anie.202104405","volume":"60","author":"M Moret","year":"2021","unstructured":"Moret M, Helmst\u00e4dter M, Grisoni F et al (2021) Beam search for automated design and scoring of novel ROR ligands with machine intelligence. Angew Chemie Int Ed 60(35):19477\u201319482","journal-title":"Angew Chemie Int Ed"},{"issue":"6","key":"646_CR33","doi-asserted-by":"publisher","first-page":"2572","DOI":"10.1021\/acs.jcim.0c01328","volume":"61","author":"J Zhang","year":"2021","unstructured":"Zhang J, Mercado R, Engkvist O, Chen H (2021) Comparative study of deep generative models on chemical space coverage. J Chem Inf Model 61(6):2572\u20132581","journal-title":"J Chem Inf Model"},{"key":"646_CR34","doi-asserted-by":"crossref","unstructured":"Flam-Shepherd D, Zhu K, Aspuru-Guzik A (2022) Language models can learn complex molecular distributions. Nat Commun 13(3293)","DOI":"10.1038\/s41467-022-30839-x"},{"key":"646_CR35","unstructured":"Cieplinski T, Danel T, Podlewska S, Jastrz\u0119bski S (2020) We should at least be able to design molecules that dock well. arXiv preprint arXiv:2006.16955"},{"key":"646_CR36","unstructured":"Huang K, Fu T, Gao W, et al (2021) Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development. arXiv preprint arXiv:2102.09548"},{"issue":"12","key":"646_CR37","doi-asserted-by":"publisher","first-page":"5699","DOI":"10.1021\/acs.jcim.0c00343","volume":"60","author":"S Amabilino","year":"2020","unstructured":"Amabilino S, Pog\u00e1ny P, Pickett SD, Green DVS (2020) Guidelines for recurrent neural network transfer learning-based molecular generation of focused libraries. J Chem Inf Model 60(12):5699\u20135713","journal-title":"J Chem Inf Model"},{"issue":"24","key":"646_CR38","doi-asserted-by":"publisher","first-page":"eabg3338","DOI":"10.1126\/sciadv.abg3338","volume":"7","author":"F Grisoni","year":"2021","unstructured":"Grisoni F, Huisman BJH, Button AL et al (2021) Combining generative artificial intelligence and on-chip synthesis for de novo drug design. Sci Adv 7(24):eabg3338","journal-title":"Sci Adv"},{"key":"646_CR39","first-page":"55","volume":"32\u201333","author":"P Renz","year":"2020","unstructured":"Renz P, Van Rompaey D, Wegner JK et al (2020) On failure modes in molecule generation and optimization. Drug Discov Today Technol 32\u201333:55\u201363","journal-title":"Drug Discov Today Technol"},{"key":"646_CR40","doi-asserted-by":"crossref","unstructured":"Langevin M, Vuilleumier R, Bianciotto M (2022) Explaining and avoiding failure modes in goal-directed generation of small molecules. J Cheminform 14(20)","DOI":"10.1186\/s13321-022-00601-y"},{"key":"646_CR41","unstructured":"Neil D, Segler M, Guasch L, et al (2018) Exploring deep recurrent models with reinforcement learning for molecule design. In: 6th International Conference on Learning Representations"},{"key":"646_CR42","unstructured":"Sutton RS, Barto AG (2018) Policy Gradient Methods. In: Reinforcement Learning: An Introduction, 2nd ed. MIT Press, p 326"},{"issue":"3","key":"646_CR43","doi-asserted-by":"publisher","first-page":"136","DOI":"10.1002\/jcc.26441","volume":"42","author":"M Tashiro","year":"2020","unstructured":"Tashiro M, Imamura Y, Katouda M (2020) De novo generation of optically active small organic molecules using Monte Carlo tree search combined with recurrent neural network. J Comput Chem 42(3):136\u2013143","journal-title":"J Comput Chem"},{"key":"646_CR44","doi-asserted-by":"crossref","unstructured":"Erikawa D, Yasuo N (2021) Sekijima M (2021) MERMAID: an open source automated hit-to-lead method based on deep reinforcement learning. J Cheminform 13(94)","DOI":"10.1186\/s13321-021-00572-6"},{"issue":"12","key":"646_CR45","doi-asserted-by":"publisher","first-page":"2658","DOI":"10.1021\/acs.jcim.0c00833","volume":"60","author":"J Boitreaud","year":"2020","unstructured":"Boitreaud J, Mallet V, Oliver C, Waldispuhl J (2020) OptiMol: optimization of binding affinities in chemical space for drug discovery. J Chem Inf Model 60(12):5658\u20135666","journal-title":"J Chem Inf Model"},{"issue":"1","key":"646_CR46","doi-asserted-by":"publisher","first-page":"22104","DOI":"10.1038\/s41598-020-78537-2","volume":"10","author":"W Jeon","year":"2020","unstructured":"Jeon W, Kim D (2020) Autonomous molecule generation using reinforcement learning and docking to develop potential novel inhibitors. Sci Rep 10(1):22104","journal-title":"Sci Rep"},{"key":"646_CR47","doi-asserted-by":"publisher","first-page":"e18","DOI":"10.7717\/peerj-pchem.18","volume":"3","author":"C Steinmann","year":"2021","unstructured":"Steinmann C, Jensen JH (2021) Using a genetic algorithm to find molecules with good docking scores. PeerJ Phys Chem 3:e18","journal-title":"PeerJ Phys Chem"},{"key":"646_CR48","doi-asserted-by":"publisher","first-page":"7079","DOI":"10.1039\/D1SC00231G","volume":"12","author":"A Nigam","year":"2021","unstructured":"Nigam A, Pollice R, Krenn M et al (2021) Beyond generative models: Superfast traversal, optimization, novelty, exploration and discovery (STONED) algorithm for molecules using SELFIES. Chem Sci 12:7079\u20137090","journal-title":"Chem Sci"},{"key":"646_CR49","doi-asserted-by":"crossref","unstructured":"Nigam A, Pollice R, Aspuru-Guzik A Parallel Tempered Genetic Algorithm Guided by Deep Neural Networks for Inverse Molecular Design. Digital Discovery 1:390\u2013404","DOI":"10.1039\/D2DD00003B"},{"issue":"11","key":"646_CR50","doi-asserted-by":"publisher","first-page":"5589","DOI":"10.1021\/acs.jcim.1c00746","volume":"61","author":"Z Xu","year":"2021","unstructured":"Xu Z, Wauchope OR, Frank AT (2021) Navigating chemical space by interfacing generative artificial intelligence and molecular docking. J Chem Inf Model 61(11):5589\u20135600","journal-title":"J Chem Inf Model"},{"issue":"7","key":"646_CR51","doi-asserted-by":"publisher","first-page":"3304","DOI":"10.1021\/acs.jcim.1c00679","volume":"61","author":"B Ma","year":"2021","unstructured":"Ma B, Terayama K, Matsumoto S et al (2021) Structure-based de novo molecular generator combined with artificial intelligence and docking simulations. J Chem Inf Model 61(7):3304\u20133313","journal-title":"J Chem Inf Model"},{"key":"646_CR52","doi-asserted-by":"crossref","unstructured":"Thomas M, Smith RT, O\u2019Boyle NM et al (2021) Comparison of structure- and ligand-based scoring functions for deep generative models: a GPCR case study. J Cheminform 13(39)","DOI":"10.1186\/s13321-021-00516-0"},{"key":"646_CR53","doi-asserted-by":"crossref","unstructured":"Guo J, Janet JP, Bauer MR et al (2021) DockStream: a docking wrapper to enhance de novo molecular design. J Cheminform 13(8)","DOI":"10.1186\/s13321-021-00563-7"},{"issue":"9","key":"646_CR54","doi-asserted-by":"publisher","first-page":"4311","DOI":"10.1021\/acs.jcim.0c00120","volume":"60","author":"P Ghanakota","year":"2020","unstructured":"Ghanakota P, Bos PH, Konze KD et al (2020) Combining cloud-based free-energy calculations, synthetically aware enumerations, and goal-directed generative machine learning for rapid large-scale chemical exploration and optimization. J Chem Inf Model 60(9):4311\u20134325","journal-title":"J Chem Inf Model"},{"issue":"2","key":"646_CR55","doi-asserted-by":"publisher","first-page":"621","DOI":"10.1021\/acs.jcim.0c01060","volume":"61","author":"SR Krishnan","year":"2021","unstructured":"Krishnan SR, Bung N, Bulusu G, Roy A (2021) Accelerating de Novo drug design against novel proteins using deep learning. J Chem Inf Model 61(2):621\u2013630","journal-title":"J Chem Inf Model"},{"issue":"2","key":"646_CR56","doi-asserted-by":"publisher","first-page":"895","DOI":"10.1021\/acs.jcim.8b00545","volume":"59","author":"M Su","year":"2019","unstructured":"Su M, Yang Q, Du Y et al (2019) Comparative assessment of scoring functions: the CASF-2016 update. J Chem Inf Model 59(2):895\u2013913","journal-title":"J Chem Inf Model"},{"issue":"14","key":"646_CR57","doi-asserted-by":"publisher","first-page":"6582","DOI":"10.1021\/jm300687e","volume":"55","author":"MM Mysinger","year":"2012","unstructured":"Mysinger MM, Carchia M, Irwin JJ, Shoichet BK (2012) Directory of useful decoys, enhanced (DUD-E): better ligands and decoys for better benchmarking. J Med Chem 55(14):6582\u20136594","journal-title":"J Med Chem"},{"issue":"12","key":"646_CR58","doi-asserted-by":"publisher","first-page":"5918","DOI":"10.1021\/acs.jcim.0c00915","volume":"60","author":"T Blaschke","year":"2020","unstructured":"Blaschke T, Ar\u00fas-Pous J, Chen H et al (2020) REINVENT 2.0: an AI tool for De Novo drug design. J Chem Inf Model 60(12):5918\u20135922","journal-title":"J Chem Inf Model"},{"issue":"9","key":"646_CR59","doi-asserted-by":"publisher","first-page":"2046","DOI":"10.1021\/acs.jcim.1c00469","volume":"62","author":"V Fialkov\u00e1","year":"2021","unstructured":"Fialkov\u00e1 V, Zhao J, Papadopoulos K et al (2021) LibINVENT: reaction-based generative scaffold decoration for in silico library design. J Chem Inf Model 62(9):2046\u20132063","journal-title":"J Chem Inf Model"},{"key":"646_CR60","unstructured":"Thomas M (2021) MolScore. In: GitHub. https:\/\/github.com\/MorganCThomas\/MolScore\/. Accessed 28 Mar 2022"},{"issue":"11","key":"646_CR61","doi-asserted-by":"publisher","first-page":"2324","DOI":"10.1021\/acs.jcim.5b00559","volume":"55","author":"T Sterling","year":"2015","unstructured":"Sterling T, Irwin JJ (2015) ZINC 15\u2014ligand discovery for everyone. J Chem Inf Model 55(11):2324\u20132337","journal-title":"J Chem Inf Model"},{"key":"646_CR62","doi-asserted-by":"publisher","first-page":"615","DOI":"10.1021\/ci960169p","volume":"37","author":"R Wang","year":"1997","unstructured":"Wang R, Fu Y, Lai L (1997) A new atom-additive method for calculating partition coefficients. J Chem Inf Comput Sci 37:615\u2013621","journal-title":"J Chem Inf Comput Sci"},{"issue":"1","key":"646_CR63","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1517\/17425255.1.1.91","volume":"1","author":"AS Kalgutkar","year":"2005","unstructured":"Kalgutkar AS, Soglia JR (2005) Minimising the potential for metabolic activation in drug discovery. Expert Opin Drug Metab Toxicol 1(1):91\u2013142","journal-title":"Expert Opin Drug Metab Toxicol"},{"issue":"3","key":"646_CR64","doi-asserted-by":"publisher","first-page":"161","DOI":"10.2174\/1389200054021799","volume":"6","author":"A Kalgutkar","year":"2005","unstructured":"Kalgutkar A, Gardner I, Obach R et al (2005) A comprehensive listing of bioactivation pathways of organic functional groups. Curr Drug Metab 6(3):161\u2013225","journal-title":"Curr Drug Metab"},{"issue":"7","key":"646_CR65","doi-asserted-by":"publisher","first-page":"2719","DOI":"10.1021\/jm901137j","volume":"53","author":"JB Baell","year":"2010","unstructured":"Baell JB, Holloway GA (2010) New substructure filters for removal of pan assay interference compounds (PAINS) from screening libraries and for their exclusion in bioassays. J Med Chem 53(7):2719\u20132740","journal-title":"J Med Chem"},{"key":"646_CR66","doi-asserted-by":"publisher","DOI":"10.26434\/CHEMRXIV.10119299.V1","author":"M Moret","year":"2019","unstructured":"Moret M, Friedrich L, Grisoni F et al (2019) Generating customized compound libraries for drug discovery with machine intelligence. ChemRxiv. https:\/\/doi.org\/10.26434\/chemrxiv.10119299.v1","journal-title":"ChemRxiv"},{"key":"646_CR67","unstructured":"BenevolentAI GuacaMol Baselines. In: GitHub. https:\/\/github.com\/BenevolentAI\/guacamol_baselines. Accessed 3 Mar 2022"},{"key":"646_CR68","unstructured":"Vaswani A, Shazeer N, Parmar N, et al (2017) Attention is all you need. In: Advances in Neural Information Processing Systems. Advances in Neural Information Processing Systems 30:5999\u20136009"},{"key":"646_CR69","unstructured":"Parisotto E, Song HF, Rae JW, et al (2020) Stabilizing transformers for reinforcement learning. Proceedings of the 37th International Conference on Machine Learning, PMLR 119:7487\u20137498"},{"key":"646_CR70","doi-asserted-by":"crossref","unstructured":"Dai Z, Yang Z, Yang Y, et al (2019) Transformer-XL: Attentive language models beyond a fixed-length context. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics 2978\u20132988","DOI":"10.18653\/v1\/P19-1285"},{"key":"646_CR71","unstructured":"Radford A, Narasimhan K, Salimans T, Sutskever I (2018) Improving Language Understanding by Generative Pre-Training"},{"key":"646_CR72","volume-title":"Reinforcement Learning: an introduction, second edi","author":"RS Sutton","year":"2018","unstructured":"Sutton RS, Barto AG (2018) Reinforcement Learning: an introduction, second edi. MIT Press"},{"key":"646_CR73","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1007\/BF00992696","volume":"8","author":"RJ Williams","year":"1992","unstructured":"Williams RJ (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach Learn 8:229\u2013256","journal-title":"Mach Learn"},{"key":"646_CR74","unstructured":"Jaques N, Gu S, Bahdanau D, et al (201) Sequence tutor: conservative fine-tuning of sequence generation models with KL-control. Proceedings of the 34th International Conference on Machine Learning, PMLR 70 4:1645\u20131654"},{"key":"646_CR75","doi-asserted-by":"crossref","unstructured":"Blaschke T, Engkvist O, Bajorath J, Chen H (2020) Memory-assisted reinforcement learning for diverse molecular de novo design. J Cheminform 12(68)","DOI":"10.1186\/s13321-020-00473-0"},{"issue":"5","key":"646_CR76","doi-asserted-by":"publisher","first-page":"742","DOI":"10.1021\/ci100050t","volume":"50","author":"D Rogers","year":"2010","unstructured":"Rogers D, Hahn M (2010) Extended-connectivity fingerprints. J Chem Inf Model 50(5):742\u2013754","journal-title":"J Chem Inf Model"},{"issue":"15","key":"646_CR77","doi-asserted-by":"publisher","first-page":"2887","DOI":"10.1021\/jm9602928","volume":"39","author":"GW Bemis","year":"1996","unstructured":"Bemis GW, Murcko MA (1996) The properties of known drugs. 1. Molecular frameworks. J Med Chem 39(15):2887\u20132893","journal-title":"J Med Chem"},{"key":"646_CR78","doi-asserted-by":"publisher","first-page":"64","DOI":"10.1021\/ci00046a002","volume":"25","author":"DH Smith","year":"1985","unstructured":"Smith DH, Carhart RE, Venkataraghavan R (1985) Atom pairs as molecular features in structure-activity studies: definition and applications. J Chem Inf Comput Sci 25:64\u201373","journal-title":"J Chem Inf Comput Sci"},{"key":"646_CR79","unstructured":"RDKit Open-source cheminformatics. http:\/\/www.rdkit.org"},{"issue":"7695","key":"646_CR80","doi-asserted-by":"publisher","first-page":"269","DOI":"10.1038\/nature25758","volume":"555","author":"S Wang","year":"2018","unstructured":"Wang S, Che T, Levit A et al (2018) Structure of the D2 dopamine receptor bound to the atypical antipsychotic drug risperidone. Nature 555(7695):269\u2013273","journal-title":"Nature"},{"issue":"7398","key":"646_CR81","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1038\/nature10954","volume":"485","author":"A Manglik","year":"2012","unstructured":"Manglik A, Kruse AC, Kobilka TS et al (2012) (2012) Crystal structure of the \u00b5-opioid receptor bound to a morphinan antagonist. Nature 485(7398):321\u2013326","journal-title":"Nat"},{"issue":"4","key":"646_CR82","doi-asserted-by":"publisher","first-page":"833","DOI":"10.1016\/j.cell.2015.04.011","volume":"161","author":"H Zhang","year":"2015","unstructured":"Zhang H, Unal H, Gati C et al (2015) Structure of the angiotensin receptor revealed by serial femtosecond crystallography. Cell 161(4):833\u2013844","journal-title":"Cell"},{"issue":"4","key":"646_CR83","doi-asserted-by":"publisher","first-page":"1528","DOI":"10.1021\/acs.jmedchem.9b01787","volume":"63","author":"M Rappas","year":"2020","unstructured":"Rappas M, Ali AAE, Bennett KA et al (2020) Comparison of orexin 1 and orexin 2 ligand binding modes using x-ray crystallography and computational analysis. J Med Chem 63(4):1528\u20131543","journal-title":"J Med Chem"},{"key":"646_CR84","unstructured":"Schr\u00f6dinger Release 2019-4 Protein Preparation Wizard"},{"issue":"12","key":"646_CR85","doi-asserted-by":"publisher","first-page":"681","DOI":"10.1007\/s10822-007-9133-z","volume":"21","author":"JC Shelley","year":"2007","unstructured":"Shelley JC, Cholleti A, Frye LL et al (2007) Epik: a software program for pKa prediction and protonation state generation for drug-like molecules. J Comput Aided Mol Des 21(12):681\u2013691","journal-title":"J Comput Aided Mol Des"},{"issue":"7","key":"646_CR86","doi-asserted-by":"publisher","first-page":"2284","DOI":"10.1021\/ct200133y","volume":"7","author":"CR Sondergaard","year":"2011","unstructured":"Sondergaard CR, Olsson MHM, Rostkowski M, Jensen JH (2011) Improved treatment of ligands and coupling effects in empirical calculation and rationalization of pKa values. J Chem Theory Comput 7(7):2284\u20132295","journal-title":"J Chem Theory Comput"},{"issue":"3","key":"646_CR87","doi-asserted-by":"publisher","first-page":"1863","DOI":"10.1021\/acs.jctc.8b01026","volume":"15","author":"K Roos","year":"2019","unstructured":"Roos K, Wu C, Damm W et al (2019) OPLS3e: extending force field coverage for drug-like small molecules. J Chem Theory Comput 15(3):1863\u20131874","journal-title":"J Chem Theory Comput"},{"key":"646_CR88","unstructured":"Schr\u00f6dinger Release 2019-4 LigPrep"},{"issue":"7","key":"646_CR89","doi-asserted-by":"publisher","first-page":"1739","DOI":"10.1021\/jm0306430","volume":"47","author":"RA Friesner","year":"2004","unstructured":"Friesner RA, Banks JL, Murphy RB et al (2004) Glide: a new approach for rapid, accurate docking and scoring. 1. method and assessment of docking accuracy. J Med Chem 47(7):1739\u20131749","journal-title":"J Med Chem"},{"key":"646_CR90","doi-asserted-by":"crossref","unstructured":"Sun J, Jeliazkova N, Chupakhin V et al (2017) ExCAPE-DB: an integrated large scale dataset facilitating Big Data analysis in chemogenomics. J Cheminform 9(17)","DOI":"10.1186\/s13321-017-0203-5"},{"issue":"6","key":"646_CR91","doi-asserted-by":"publisher","first-page":"598","DOI":"10.1002\/qsar.200290002","volume":"21","author":"M Ashton","year":"2002","unstructured":"Ashton M, Barnard J, Casset F et al (2002) Identification of diverse database subsets using property-based and fragment-based molecular descriptions. Quant Struct Relationships 21(6):598\u2013604","journal-title":"Quant Struct Relationships"},{"key":"646_CR92","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa F, Varoquaux G, Gramfort A et al (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12:2825\u20132830","journal-title":"J Mach Learn Res"},{"issue":"6","key":"646_CR93","doi-asserted-by":"publisher","first-page":"2623","DOI":"10.1021\/acs.jcim.1c00160","volume":"61","author":"C Esposito","year":"2021","unstructured":"Esposito C, Landrum GA, Schneider N et al (2021) GHOST: adjusting the decision threshold to handle imbalanced data in machine learning. J Chem Inf Model 61(6):2623\u20132640","journal-title":"J Chem Inf Model"},{"issue":"2","key":"646_CR94","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1038\/nchem.1243","volume":"4","author":"GR Bickerton","year":"2012","unstructured":"Bickerton GR, Paolini GV, Besnard J et al (2012) Quantifying the chemical beauty of drugs. Nat Chem 4(2):90\u201398","journal-title":"Nat Chem"},{"key":"646_CR95","unstructured":"You J, Liu B, Ying R, et al (2018) Graph convolutional policy network for goal-directed molecular graph generation. Advances in neural information processing systems 31"},{"key":"646_CR96","unstructured":"Jin W, Barzilay R, Jaakkola T (2020) Multi-Objective Molecule Generation using Interpretable Substructures. Proceedings of the 37th International Conference on Machine Learning, PMLR 119:4849\u20134859"},{"issue":"1","key":"646_CR97","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1021\/ci020055f","volume":"43","author":"Y Pan","year":"2003","unstructured":"Pan Y, Huang N, Cho S, MacKerell AD (2003) Consideration of molecular weight during compound selection in virtual target-based database screening. J Chem Inf Comput Sci 43(1):267\u2013272","journal-title":"J Chem Inf Comput Sci"},{"issue":"9","key":"646_CR98","doi-asserted-by":"publisher","first-page":"1736","DOI":"10.1021\/acs.jcim.8b00234","volume":"58","author":"K Preuer","year":"2018","unstructured":"Preuer K, Renz P, Unterthiner T et al (2018) Fr\u00e9chet ChemNet distance: a metric for generative models for molecules in drug discovery. J Chem Inf Model 58(9):1736\u20131741","journal-title":"J Chem Inf Model"},{"key":"646_CR99","doi-asserted-by":"crossref","unstructured":"Ar\u00fas-Pous J, Johansson SV, Prykhodko O et al (2019) Randomized SMILES strings improve the quality of molecular generative models. J Cheminform 11(71)","DOI":"10.1186\/s13321-019-0393-0"},{"issue":"8","key":"646_CR100","doi-asserted-by":"publisher","first-page":"3784","DOI":"10.1021\/acs.jmedchem.8b00836","volume":"62","author":"M Vass","year":"2019","unstructured":"Vass M, Podlewska S, De Esch IJP et al (2019) Aminergic GPCR-ligand interactions: a chemical and structural map of receptor mutation data. J Med Chem 62(8):3784\u20133839","journal-title":"J Med Chem"},{"key":"646_CR101","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1016\/j.coph.2016.07.007","volume":"30","author":"M Vass","year":"2016","unstructured":"Vass M, Kooistra AJ, Ritschel T et al (2016) Molecular interaction fingerprint approaches for GPCR drug discovery. Curr Opin Pharmacol 30:59\u201368","journal-title":"Curr Opin Pharmacol"},{"issue":"7","key":"646_CR102","doi-asserted-by":"publisher","first-page":"718","DOI":"10.1002\/cmdc.201500599","volume":"11","author":"AA Kaczor","year":"2016","unstructured":"Kaczor AA, Silva AG, Loza MI et al (2016) Structure-based virtual screening for dopamine D2 receptor ligands as potential antipsychotics. ChemMedChem 11(7):718\u2013729","journal-title":"ChemMedChem"},{"key":"646_CR103","doi-asserted-by":"crossref","unstructured":"Khemchandani Y, O\u2019Hagan S, Samanta S et al (2020) DeepGraphMolGen, a multi-objective, computational strategy for generating molecules with desirable properties: a graph convolution and reinforcement learning approach. J Cheminform 12(53)","DOI":"10.1186\/s13321-020-00454-3"},{"issue":"49","key":"646_CR104","doi-asserted-by":"publisher","first-page":"33864","DOI":"10.1021\/acsomega.1c05145","volume":"6","author":"L Yang","year":"2021","unstructured":"Yang L, Yang G, Bing Z et al (2021) Transformer-based generative model accelerating the development of novel BRAF inhibitors. ACS Omega 6(49):33864\u201333873","journal-title":"ACS Omega"},{"key":"646_CR105","doi-asserted-by":"publisher","DOI":"10.26434\/CHEMRXIV-2021-PX6KZ","author":"X Liu","year":"2021","unstructured":"Liu X, Ye K, van Vlijmen HWT et al (2021) DrugEx v3: scaffold-constrained drug design with graph transformer-based reinforcement learning. ChemRxiv preprint\u00a010.26434\/chemrxiv-2021-px6kz","journal-title":"ChemRxiv."},{"key":"646_CR106","unstructured":"Hutchins D, Schlag I, Wu Y, et al (2022) Block-recurrent transformers. arXiv preprint arXiv:2203.07852"},{"key":"646_CR107","unstructured":"Gao W, Fu T, Sun J, Coley CW (2022) Sample efficiency matters: a benchmark for practical molecular optimization. arXiv preprint arXiv:2206.12411"},{"key":"646_CR108","doi-asserted-by":"crossref","unstructured":"Korshunova M, Huang N, Capuzzi S, et al (2021) A Bag of Tricks for Automated De Novo Design of Molecules with the Desired Properties: Application to EGFR Inhibitor Discovery. ChemRxiv preprint 10.26434\/chemrxiv.14045072.v1","DOI":"10.26434\/chemrxiv.14045072"},{"key":"646_CR109","unstructured":"Patronov A, Margreitter C, Blaschke T, Guo J (2021) REINVENT 3.0. In: GitHub. https:\/\/github.com\/MolecularAI\/Reinvent\/tree\/reinvent.3.0. Accessed 28 Mar 2022"},{"key":"646_CR110","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1038\/s42256-022-00494-4","volume":"4","author":"J Guo","year":"2022","unstructured":"Guo J, Fialkov\u00e1 V, Arango JD et al (2022) Improving de novo molecular design with curriculum learning. Nat Mach Intell 4:555\u2013563","journal-title":"Nat Mach Intell"}],"container-title":["Journal of Cheminformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13321-022-00646-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13321-022-00646-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13321-022-00646-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,3]],"date-time":"2022-10-03T12:07:44Z","timestamp":1664798864000},"score":1,"resource":{"primary":{"URL":"https:\/\/jcheminf.biomedcentral.com\/articles\/10.1186\/s13321-022-00646-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,3]]},"references-count":110,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["646"],"URL":"https:\/\/doi.org\/10.1186\/s13321-022-00646-z","relation":{"has-preprint":[{"id-type":"doi","id":"10.26434\/chemrxiv-2022-prz2r","asserted-by":"object"}]},"ISSN":["1758-2946"],"issn-type":[{"value":"1758-2946","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,3]]},"assertion":[{"value":"14 April 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 September 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 October 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"68"}}