{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,31]],"date-time":"2026-01-31T04:48:34Z","timestamp":1769834914113,"version":"3.49.0"},"reference-count":44,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2020,12,22]],"date-time":"2020-12-22T00:00:00Z","timestamp":1608595200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"SipingSoft Co. Ltd"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,7,20]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>The size and quality of chemical libraries to the drug discovery pipeline are crucial for developing new drugs or repurposing existing drugs. Existing techniques such as combinatorial organic synthesis and high-throughput screening usually make the process extraordinarily tough and complicated since the search space of synthetically feasible drugs is exorbitantly huge. While reinforcement learning has been mostly exploited in the literature for generating novel compounds, the requirement of designing a reward function that succinctly represents the learning objective could prove daunting in certain complex domains. Generative adversarial network-based methods also mostly discard the discriminator after training and could be hard to train. In this study, we propose a framework for training a compound generator and learn a transferable reward function based on the entropy maximization inverse reinforcement learning (IRL) paradigm. We show from our experiments that the IRL route offers a rational alternative for generating chemical compounds in domains where reward function engineering may be less appealing or impossible while data exhibiting the desired objective is readily available.<\/jats:p>","DOI":"10.1093\/bib\/bbaa364","type":"journal-article","created":{"date-parts":[[2020,11,11]],"date-time":"2020-11-11T12:09:45Z","timestamp":1605096585000},"source":"Crossref","is-referenced-by-count":11,"title":["Deep inverse reinforcement learning for structural evolution of small molecules"],"prefix":"10.1093","volume":"22","author":[{"given":"Brighter","family":"Agyemang","sequence":"first","affiliation":[{"name":"University of Electronic Science and Technology of China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wei-Ping","family":"Wu","sequence":"additional","affiliation":[{"name":"UESTC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daniel","family":"Addo","sequence":"additional","affiliation":[{"name":"UESTC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael Y","family":"Kpiebaareh","sequence":"additional","affiliation":[{"name":"UESTC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ebenezer","family":"Nanor","sequence":"additional","affiliation":[{"name":"Sipingsoft Co. Ltd, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Charles","family":"Roland Haruna","sequence":"additional","affiliation":[{"name":"UESTC"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2020,12,22]]},"reference":[{"issue":"1","key":"2021072117015385200_ref1","first-page":"1","article-title":"Molecular de-novo design through deep reinforcement learning","volume":"9","author":"Olivecrona","year":"2017","journal-title":"J Chem"},{"issue":"4","key":"2021072117015385200_ref2","doi-asserted-by":"crossref","first-page":"401","DOI":"10.1080\/13543776.2017.1272573","article-title":"Small-molecule inhibitors of hepatitis c virus (HCV) non-structural protein 5a (ns5a): a patent review (2010\u20132015)","volume":"27","author":"Ivanenkov","year":"2017","journal-title":"Expert Opin Ther Pat"},{"issue":"1","key":"2021072117015385200_ref3","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1021\/acscentsci.7b00512","article-title":"Generating focused molecule libraries for drug discovery with recurrent neural networks","volume":"4","author":"Segler","year":"2018","journal-title":"ACS Cent Sci"},{"issue":"8","key":"2021072117015385200_ref4","doi-asserted-by":"crossref","first-page":"675","DOI":"10.1007\/s10822-013-9672-4","article-title":"Estimation of the size of drug-like chemical space based on GDB-17 data","volume":"27","author":"Polishchuk","year":"2013","journal-title":"J Comput Aided Mol Des"},{"issue":"1","key":"2021072117015385200_ref5","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-019-54849-w","article-title":"Improving the odds of drug development success through human genomics: modelling study","volume":"9","author":"Hingorani","year":"2019","journal-title":"Sci Rep"},{"key":"2021072117015385200_ref6","first-page":"1","article-title":"Neural machine translation by jointly learning to align and translate","author":"Bahdanau","year":"2014","journal-title":"arXiv"},{"key":"2021072117015385200_ref7","article-title":"Language models are unsupervised multitask learners","author":"Radford","year":"2018"},{"key":"2021072117015385200_ref8","first-page":"2048","article-title":"Show, attend and tell: neural image caption generation with visual attention","volume-title":"Proceedings of the 32nd International Conference on Machine Learning","author":"Xu","year":"2015"},{"issue":"4","key":"2021072117015385200_ref9","doi-asserted-by":"crossref","first-page":"942","DOI":"10.1021\/acs.jcim.6b00740","article-title":"Protein-ligand scoring with convolutional neural networks","volume":"57","author":"Ragoza","year":"2017","journal-title":"J Chem Inf Model"},{"key":"2021072117015385200_ref10","doi-asserted-by":"crossref","first-page":"103547","DOI":"10.1016\/j.jbi.2020.103547","article-title":"Multi-view self-attention for interpretable drug-target interaction prediction","volume":"110","author":"Agyemang","year":"2020","journal-title":"J Biomed Inform"},{"issue":"2","key":"2021072117015385200_ref11","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1093\/bioinformatics\/bty535","article-title":"Compound-protein interaction prediction with end-to-end learning of neural networks for graphs and sequences","volume":"35","author":"Tsubaki","year":"2019","journal-title":"Bioinformatics"},{"issue":"25","key":"2021072117015385200_ref12","doi-asserted-by":"crossref","first-page":"6118","DOI":"10.1002\/chem.201604556","article-title":"Modelling chemical reasoning to predict and invent reactions","volume":"23","author":"Segler","year":"2017","journal-title":"Chem A Eur J"},{"issue":"1","key":"2021072117015385200_ref13","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1021\/ci00057a005","article-title":"SMILES, a chemical language and information system: 1: introduction to methodology and encoding rules","volume":"28","author":"Weininger","year":"1988","journal-title":"J Chem Inf Comput Sci"},{"key":"2021072117015385200_ref14","first-page":"2852","author":"Yu","year":"2017","journal-title":"Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence"},{"issue":"7","key":"2021072117015385200_ref15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1126\/sciadv.aap7885","article-title":"Deep reinforcement learning for de novo drug design","volume":"4","author":"Popova","year":"2018","journal-title":"Sci Adv"},{"key":"2021072117015385200_ref16","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1007\/BF00992696","article-title":"Simple statistical gradient-following methods for connectionist reinforcement learning","volume":"8","author":"Williams","year":"1992","journal-title":"Mach Learn"},{"key":"2021072117015385200_ref17","doi-asserted-by":"crossref","first-page":"548","DOI":"10.1109\/ICDM.2019.00065","article-title":"Reinforced molecule generation with heterogeneous states","volume-title":"IEEE International Conference on Data Mining (ICDM)","author":"Shi","year":"2019"},{"key":"2021072117015385200_ref18","first-page":"1","article-title":"Generalization in generation: a closer look at exposure bias","volume-title":"CoRR","author":"Schmidt","year":"2019"},{"key":"2021072117015385200_ref19","first-page":"1","article-title":"Optimizing distributions over molecular space. An objective-reinforced generative adversarial network for inverse-design chemistry (ORGANIC)","author":"Sanchez-Lengeling","year":"2017","journal-title":"ChemRxiv"},{"key":"2021072117015385200_ref20","first-page":"1","article-title":"Objective-reinforced generative adversarial networks (ORGAN) for sequence generation models","author":"Guimaraes","year":"2017","journal-title":"CoRR"},{"key":"2021072117015385200_ref21","first-page":"1","article-title":"ChemGAN challenge for drug discovery: can AI reproduce natural chemical diversity","author":"Benhenda","year":"2017","journal-title":"arXiv"},{"issue":"6","key":"2021072117015385200_ref22","doi-asserted-by":"crossref","first-page":"1194","DOI":"10.1021\/acs.jcim.7b00690","article-title":"Reinforced adversarial neural computer for de novo molecular design","volume":"58","author":"Putin","year":"2018","journal-title":"J Chem Inf Model"},{"key":"2021072117015385200_ref23","article-title":"Towards principled methods for training generative adversarial networks","author":"Arjovsky","year":"2017"},{"key":"2021072117015385200_ref24","first-page":"49","article-title":"Guided cost learning: deep inverse optimal control via policy optimization","volume-title":"Proceedings of the 33rd International Conference on Machine Learning (ICML)","author":"Finn","year":"2016"},{"key":"2021072117015385200_ref25","first-page":"1","article-title":"A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models","author":"Finn","year":"2016","journal-title":"arXiv"},{"key":"2021072117015385200_ref26","article-title":"High-dimensional continuous control using generalized advantage estimation","volume-title":"4th International Conference on Learning Representations, {ICLR}","author":"Schulman","year":"2016"},{"key":"2021072117015385200_ref27","first-page":"1","article-title":"A brief survey of deep reinforcement learning","author":"Arulkumaran","year":"2017","journal-title":"arXiv"},{"key":"2021072117015385200_ref28","first-page":"1","article-title":"Proximal policy optimization algorithms","author":"Schulman","year":"2017","journal-title":"CoRR"},{"issue":"7540","key":"2021072117015385200_ref29","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"2021072117015385200_ref30","first-page":"1889","article-title":"Trust Region Policy Optimization","volume":"37","author":"Schulman","year":"2015","journal-title":"Proceedings of the 32nd International Conference on Machine Learning"},{"key":"2021072117015385200_ref31","first-page":"1","article-title":"A survey of inverse reinforcement learning: challenges, methods and progress","author":"Arora","year":"2018","journal-title":"arXiv"},{"key":"2021072117015385200_ref32","first-page":"663","article-title":"Algorithms for inverse reinforcement learning","volume-title":"Proceedings of the Seventeenth International Conference on Machine Learning","author":"Ng","year":"2000"},{"key":"2021072117015385200_ref33","first-page":"1433","article-title":"Maximum entropy inverse reinforcement learning","volume-title":"AAAI Conference on Artificial Intelligence","author":"Ziebart","year":"2008"},{"key":"2021072117015385200_ref34","first-page":"1","article-title":"Maximum entropy deep inverse reinforcement learning","author":"Wulfmeier","year":"2016","journal-title":"arXiv"},{"key":"2021072117015385200_ref35","first-page":"4565","article-title":"Generative adversarial imitation learning","volume-title":"Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems","author":"Ho","year":"2016"},{"key":"2021072117015385200_ref36","doi-asserted-by":"crossref","first-page":"D1035","DOI":"10.1093\/nar\/gkq1126","article-title":"DrugBank 3.0: a comprehensive resource for \u2018Omics\u2019 research on drugs","volume":"39","author":"Knox","year":"2010","journal-title":"Nucleic Acids Res"},{"issue":"D1","key":"2021072117015385200_ref37","doi-asserted-by":"crossref","first-page":"D109","DOI":"10.1093\/nar\/gkr988","article-title":"KEGG for integration and interpretation of large-scale molecular data sets","volume":"40","author":"Kanehisa","year":"2011","journal-title":"Nucleic Acids Res"},{"key":"2021072117015385200_ref38","doi-asserted-by":"crossref","first-page":"D380","DOI":"10.1093\/nar\/gkv1277","article-title":"STITCH 5: augmenting protein\u2013chemical interaction networks with tissue and affinity data","volume":"44","author":"Szklarczyk","year":"2016","journal-title":"Nucleic Acids Res"},{"key":"2021072117015385200_ref39","doi-asserted-by":"crossref","first-page":"D1083","DOI":"10.1093\/nar\/gkt1031","article-title":"The ChEMBL bioactivity database: an update","volume":"42","author":"Bento","year":"2014","journal-title":"Nucleic Acids Res"},{"key":"2021072117015385200_ref40","first-page":"190","article-title":"Inferring algorithmic patterns with stack-augmented recurrent nets","volume-title":"Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems","author":"Joulin","year":"2015"},{"key":"2021072117015385200_ref41","article-title":"ExCAPE-DB: an integrated large scale dataset facilitating big data analysis in chemogenomics","volume":"9","author":"Sun","year":"2017","journal-title":"J Chem"},{"issue":"1","key":"2021072117015385200_ref42","first-page":"1","article-title":"Naples: a natural products likeness scorer-web application and database","volume":"11","author":"Sorokina","year":"2019","journal-title":"J Chem"},{"key":"2021072117015385200_ref43","article-title":"Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions","volume":"1","author":"Ertl","year":"2009","journal-title":"J Chem"},{"key":"2021072117015385200_ref44","first-page":"1","article-title":"Stabilizing transformers for reinforcement learning","author":"Parisotto","year":"2019","journal-title":"CoRR"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/bib\/article-pdf\/22\/4\/bbaa364\/39136712\/bbaa364.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/bib\/article-pdf\/22\/4\/bbaa364\/39136712\/bbaa364.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,7,21]],"date-time":"2021-07-21T17:19:31Z","timestamp":1626887971000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbaa364\/6043289"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12,22]]},"references-count":44,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,7,20]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbaa364","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,7]]},"published":{"date-parts":[[2020,12,22]]},"article-number":"bbaa364"}}