{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T08:11:00Z","timestamp":1786090260188,"version":"3.56.0"},"publisher-location":"Cham","reference-count":29,"publisher":"Springer International Publishing","isbn-type":[{"value":"9783030304928","type":"print"},{"value":"9783030304935","type":"electronic"}],"license":[{"start":{"date-parts":[[2019,1,1]],"date-time":"2019-01-01T00:00:00Z","timestamp":1546300800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We describe a Transformer model for a retrosynthetic reaction prediction task. The model is trained on 45\u00a0033 experimental reaction examples extracted from USA patents. It can successfully predict the reactants set for 42.7% of cases on the external test set. During the training procedure, we applied different learning rate schedules and snapshot learning. These techniques can prevent overfitting and thus can be a reason to get rid of internal validation dataset that is advantageous for deep models with millions of parameters. We thoroughly investigated different approaches to train Transformer models and found that snapshot learning with averaging weights on learning rates minima works best. While decoding the model output probabilities there is a strong influence of the temperature that improves at <jats:inline-formula><jats:tex-math>$$\\text {T}=1.3$$<\/jats:tex-math><\/jats:inline-formula> the accuracy of models up\u00a0to 1\u20132%.<\/jats:p>","DOI":"10.1007\/978-3-030-30493-5_78","type":"book-chapter","created":{"date-parts":[[2019,9,10]],"date-time":"2019-09-10T20:03:41Z","timestamp":1568145821000},"page":"817-830","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":98,"title":["A Transformer Model for Retrosynthesis"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4786-9806","authenticated-orcid":false,"given":"Pavel","family":"Karpov","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9828-386X","authenticated-orcid":false,"given":"Guillaume","family":"Godin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6855-0012","authenticated-orcid":false,"given":"Igor V.","family":"Tetko","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2019,9,9]]},"reference":[{"key":"78_CR1","unstructured":"Abadi, M., et al.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). https:\/\/www.tensorflow.org\/"},{"issue":"11","key":"78_CR2","doi-asserted-by":"publisher","first-page":"1127","DOI":"10.1070\/RCR4746","volume":"86","author":"II Baskin","year":"2017","unstructured":"Baskin, I.I., Madzhidov, T.I., Antipin, I.S., Varnek, A.A.: Artificial intelligence in synthetic chemistry: achievements and prospects. Russ. Chem. Rev. 86(11), 1127\u20131156 (2017). https:\/\/doi.org\/10.1070\/RCR4746","journal-title":"Russ. Chem. Rev."},{"issue":"8","key":"78_CR3","doi-asserted-by":"publisher","first-page":"785","DOI":"10.1080\/17460441.2016.1201262","volume":"11","author":"II Baskin","year":"2016","unstructured":"Baskin, I.I., Winkler, D., Tetko, I.V.: A renaissance of neural networks in drug discovery. Expert Opin. Drug Discov. 11(8), 785\u2013795 (2016). https:\/\/doi.org\/10.1080\/17460441.2016.1201262","journal-title":"Expert Opin. Drug Discov."},{"issue":"2","key":"78_CR4","doi-asserted-by":"publisher","first-page":"673","DOI":"10.1021\/acs.jcim.8b00801","volume":"59","author":"JL Baylon","year":"2019","unstructured":"Baylon, J.L., Cilfone, N.A., Gulcher, J.R., Chittenden, T.W.: Enhancing retrosynthetic reaction prediction with deep learning using multiscale reaction classification. J. Chem. Inf. Model. 59(2), 673\u2013688 (2019). https:\/\/doi.org\/10.1021\/acs.jcim.8b00801","journal-title":"J. Chem. Inf. Model."},{"issue":"6","key":"78_CR5","doi-asserted-by":"publisher","first-page":"1241","DOI":"10.1016\/j.drudis.2018.01.039","volume":"23","author":"H Chen","year":"2018","unstructured":"Chen, H., Engkvist, O., Wang, Y., Olivecrona, M., Blaschke, T.: The rise of deep learning in drug discovery. Drug Discov. Today 23(6), 1241\u20131250 (2018). https:\/\/doi.org\/10.1016\/j.drudis.2018.01.039","journal-title":"Drug Discov. Today"},{"issue":"5","key":"78_CR6","doi-asserted-by":"publisher","first-page":"1281","DOI":"10.1021\/acs.accounts.8b00087","volume":"51","author":"CW Coley","year":"2018","unstructured":"Coley, C.W., Green, W.H., Jensen, K.F.: Machine learning in computer-aided synthesis planning. Acc. Chem. Res. 51(5), 1281\u20131289 (2018). https:\/\/doi.org\/10.1021\/acs.accounts.8b00087","journal-title":"Acc. Chem. Res."},{"issue":"2","key":"78_CR7","doi-asserted-by":"publisher","first-page":"252","DOI":"10.1021\/acs.jcim.7b00622","volume":"58","author":"CW Coley","year":"2018","unstructured":"Coley, C.W., Rogers, L., Green, W.H., Jensen, K.F.: SCScore: synthetic complexity learned from a reaction corpus. J. Chem. Inf. Model. 58(2), 252\u2013261 (2018). https:\/\/doi.org\/10.1021\/acs.jcim.7b00622","journal-title":"J. Chem. Inf. Model."},{"key":"78_CR8","volume-title":"The Logic of Chemical Synthesis","author":"EJ Corey","year":"1995","unstructured":"Corey, E.J., Cheng, X.M.: The Logic of Chemical Synthesis. Wiley, Hoboken (1995)"},{"issue":"6","key":"78_CR9","doi-asserted-by":"publisher","first-page":"1203","DOI":"10.1016\/j.drudis.2018.02.014","volume":"23","author":"O Engkvist","year":"2018","unstructured":"Engkvist, O., et al.: Computational prediction of chemical reactions: current status and outlook. Drug Discov. Today 23(6), 1203\u20131218 (2018). https:\/\/doi.org\/10.1016\/j.drudis.2018.02.014","journal-title":"Drug Discov. Today"},{"key":"78_CR10","unstructured":"Ertl, P., Lewis, R., Martin, E., Polyakov, V.: In silico generation of novel, drug-like chemical matter using the LSTM neural network. arXiv (2017). arXiv:1712.07449"},{"issue":"1","key":"78_CR11","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1186\/1758-2946-1-8","volume":"1","author":"P Ertl","year":"2009","unstructured":"Ertl, P., Schuffenhauer, A.: Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. J. Cheminform. 1(1), 8 (2009). https:\/\/doi.org\/10.1186\/1758-2946-1-8","journal-title":"J. Cheminform."},{"key":"78_CR12","doi-asserted-by":"publisher","first-page":"1800104","DOI":"10.1002\/minf.201800104","volume":"37","author":"T Gimadiev","year":"2018","unstructured":"Gimadiev, T., et al.: Bimolecular nucleophilic substitution reactions: predictive models for rate constants and molecular reaction pairs analysis. Mol. Inform. 37, 1800104 (2018). https:\/\/doi.org\/10.1002\/minf.201800104","journal-title":"Mol. Inform."},{"issue":"2","key":"78_CR13","doi-asserted-by":"publisher","first-page":"268","DOI":"10.1021\/acscentsci.7b00572","volume":"4","author":"R G\u00f3mez-Bombarelli","year":"2018","unstructured":"G\u00f3mez-Bombarelli, R., et al.: Automatic chemical design using a data-driven continuous representation of molecules. ACS Cent. Sci. 4(2), 268\u2013276 (2018). https:\/\/doi.org\/10.1021\/acscentsci.7b00572","journal-title":"ACS Cent. Sci."},{"key":"78_CR14","unstructured":"Guimaraes, G.L., Sanchez-Lengeling, B., Outeiral, C., Farias, P.L.C., Aspuru-Guzik, A.: Objective-reinforced generative adversarial networks (ORGAN) for sequence generation models. arXiv (2017). arXiv:1705.10843"},{"key":"78_CR15","unstructured":"Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv (2015). arXiv:1503.02531"},{"key":"78_CR16","unstructured":"Huang, G., Li, Y., Pleiss, G., Liu, Z., Hopcroft, J.E., Weinberger, K.Q.: Snapshot ensembles: train 1, get M for free. arXiv (2017). arXiv:1704.00109"},{"key":"78_CR17","unstructured":"Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., Wilson, A.G.: Averaging weights leads to wider optima and better generalization. arXiv (2018). arXiv:1803.05407"},{"key":"78_CR18","unstructured":"Kimber, T.B., Engelke, S., Tetko, I.V., Bruno, E., Godin, G.: Synergy effect between convolutional neural networks and the multiplicity of SMILES for improvement of molecular prediction. arXiv (2018). arXiv:1812.04439"},{"issue":"2","key":"78_CR19","doi-asserted-by":"publisher","first-page":"299","DOI":"10.1134\/S0012500811100107","volume":"440","author":"AA Kravtsov","year":"2011","unstructured":"Kravtsov, A.A., Karpov, P.V., Baskin, I.I., Palyulin, V.A., Zefirov, N.S.: Prediction of rate constants of SN2 reactions by the multicomponent QSPR method. Dokl. Chem. 440(2), 299\u2013301 (2011). https:\/\/doi.org\/10.1134\/S0012500811100107","journal-title":"Dokl. Chem."},{"issue":"3","key":"78_CR20","doi-asserted-by":"publisher","first-page":"593","DOI":"10.1021\/ci800228y","volume":"49","author":"J Law","year":"2009","unstructured":"Law, J., et al.: Route designer: a retrosynthetic analysis tool utilizing automated retrosynthetic rule generation. J. Chem. Inf. Model. 49(3), 593\u2013602 (2009). https:\/\/doi.org\/10.1021\/ci800228y","journal-title":"J. Chem. Inf. Model."},{"issue":"10","key":"78_CR21","doi-asserted-by":"publisher","first-page":"1103","DOI":"10.1021\/acscentsci.7b00303","volume":"3","author":"B Liu","year":"2017","unstructured":"Liu, B., et al.: Retrosynthetic reaction prediction using neural sequence-to-sequence models. ACS Cent. Sci. 3(10), 1103\u20131113 (2017). https:\/\/doi.org\/10.1021\/acscentsci.7b00303","journal-title":"ACS Cent. Sci."},{"key":"78_CR22","unstructured":"Lowe, D.M.: Extraction of chemical structures and reactions from the literature. Ph.D. thesis, Pembroke College (2012). https:\/\/www.repository.cam.ac.uk\/handle\/1810\/244727"},{"issue":"48","key":"78_CR23","doi-asserted-by":"publisher","first-page":"1758","DOI":"10.1186\/s13321-017-0235-x","volume":"9","author":"M Olivecrona","year":"2017","unstructured":"Olivecrona, M., Blaschke, T., hongming Chen, O.E.: Molecular de-novo design through deep reinforcement learning. J Cheminform. 9(48), 1758\u20132946 (2017). https:\/\/doi.org\/10.1186\/s13321-017-0235-x","journal-title":"J Cheminform."},{"issue":"1","key":"78_CR24","doi-asserted-by":"publisher","first-page":"43","DOI":"10.2478\/pralin-2018-0002","volume":"110","author":"Martin Popel","year":"2018","unstructured":"Popel, M., Bojar, O.: Training tips for the transformer model. arXiv (2018). https:\/\/doi.org\/10.2478\/pralin-2018-0002","journal-title":"The Prague Bulletin of Mathematical Linguistics"},{"issue":"12","key":"78_CR25","doi-asserted-by":"publisher","first-page":"2336","DOI":"10.1021\/acs.jcim.6b00564","volume":"56","author":"N Schneider","year":"2016","unstructured":"Schneider, N., Stiefl, N., Landrum, G.A.: What\u2019s what: the (nearly) definitive guide to reaction role assignment. J. Chem. Inf. Model. 56(12), 2336\u20132346 (2016). https:\/\/doi.org\/10.1021\/acs.jcim.6b00564","journal-title":"J. Chem. Inf. Model."},{"key":"78_CR26","doi-asserted-by":"crossref","unstructured":"Schwaller, P., Gaudin, T., Lanyi, D., Bekas, C., Laino, T.: Found in translation: predicting outcomes of complex organic chemistry reactions using neural sequence-to-sequence models. arXiv (2018). arXiv:1711.04810","DOI":"10.1039\/C8SC02339E"},{"key":"78_CR27","doi-asserted-by":"crossref","unstructured":"Schwaller, P., Laino, T., Gaudin, T., Bolgar, P., Bekas, C., Lee, A.A.: Molecular transformer for chemical reaction prediction and uncertainty estimation. arXiv (2018). arXiv:1811.02633","DOI":"10.26434\/chemrxiv.7297379.v1"},{"key":"78_CR28","doi-asserted-by":"publisher","first-page":"604","DOI":"10.1038\/nature25978","volume":"555","author":"MH Segler","year":"2018","unstructured":"Segler, M.H., Preuss, M., Waller, M.P.: Planning chemical synthesis with deep neural networks and symbolic AI. Nature 555, 604\u2013610 (2018). https:\/\/doi.org\/10.1038\/nature25978","journal-title":"Nature"},{"key":"78_CR29","unstructured":"Vaswani, A., et al.: Attention is all you need. arXiv (2017). arXiv:1706.03762"}],"container-title":["Lecture Notes in Computer Science","Artificial Neural Networks and Machine Learning \u2013 ICANN 2019: Workshop and Special Sessions"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-030-30493-5_78","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,13]],"date-time":"2024-03-13T16:56:03Z","timestamp":1710348963000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-030-30493-5_78"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019]]},"ISBN":["9783030304928","9783030304935"],"references-count":29,"URL":"https:\/\/doi.org\/10.1007\/978-3-030-30493-5_78","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"value":"0302-9743","type":"print"},{"value":"1611-3349","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019]]},"assertion":[{"value":"9 September 2019","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"ICANN","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Artificial Neural Networks","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Munich","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Germany","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2019","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"17 September 2019","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"19 September 2019","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"28","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"icann2019","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/e-nns.org\/icann2019\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}