{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,17]],"date-time":"2026-02-17T02:45:56Z","timestamp":1771296356522,"version":"3.50.1"},"publisher-location":"Cham","reference-count":28,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783031723803","type":"print"},{"value":"9783031723810","type":"electronic"}],"license":[{"start":{"date-parts":[[2024,9,20]],"date-time":"2024-09-20T00:00:00Z","timestamp":1726790400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,9,20]],"date-time":"2024-09-20T00:00:00Z","timestamp":1726790400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This study explores the impact of pretraining Graph Transformers using atom-level quantum-mechanical features for molecular property modeling. We utilize the ADMET Therapeutic Data Commons datasets to evaluate the benefits of this approach. Our results show that pretraining on quantum atomic properties improves the performance of the Graphormer model. We conduct comparisons with two other pretraining strategies: one based on molecular quantum properties (specifically the HOMO-LUMO gap) and another using a self-supervised atom masking technique. Additionally, we employ a spectral analysis of Attention Rollout matrices to understand the underlying reasons for these performance enhancements. Our findings suggest that models pretrained on atom-level quantum mechanics are better at capturing low-frequency Laplacian eigenmodes from the molecular graphs, which correlates with improved outcomes on most evaluated downstream tasks, as measured by our custom metric.<\/jats:p>","DOI":"10.1007\/978-3-031-72381-0_7","type":"book-chapter","created":{"date-parts":[[2024,9,19]],"date-time":"2024-09-19T13:10:16Z","timestamp":1726751416000},"page":"71-81","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Atom-Level Quantum Pretraining Enhances the\u00a0Spectral Perception of\u00a0Molecular Graphs in\u00a0Graphormer"],"prefix":"10.1007","author":[{"given":"Alessio","family":"Fallani","sequence":"first","affiliation":[]},{"given":"Jos\u00e9","family":"Arjona-Medina","sequence":"additional","affiliation":[]},{"given":"Konstantin","family":"Chernichenko","sequence":"additional","affiliation":[]},{"given":"Ramil","family":"Nugmanov","sequence":"additional","affiliation":[]},{"given":"J\u00f6rg Kurt","family":"Wegner","sequence":"additional","affiliation":[]},{"given":"Alexandre","family":"Tkatchenko","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,9,20]]},"reference":[{"key":"7_CR1","doi-asserted-by":"crossref","unstructured":"Abnar, S., Zuidema, W.H.: Quantifying attention flow in transformers (2020). https:\/\/arxiv.org\/abs\/2005.00928","DOI":"10.18653\/v1\/2020.acl-main.385"},{"key":"7_CR2","doi-asserted-by":"publisher","unstructured":"Beck, M.E.: Do fukui function maxima relate to sites of metabolism? A critical case study. J. Chem. Inform. Model. 45(2), 273\u2013282 (2005). https:\/\/doi.org\/10.1021\/ci049687n, pMID: 15807488","DOI":"10.1021\/ci049687n"},{"key":"7_CR3","doi-asserted-by":"publisher","unstructured":"Born, J., et al.: Chemical representation learning for toxicity prediction. Digit. Disc. 2, 674\u2013691 (2023). https:\/\/doi.org\/10.1039\/D2DD00099G","DOI":"10.1039\/D2DD00099G"},{"key":"7_CR4","doi-asserted-by":"publisher","unstructured":"Broccatelli, F., Trager, R., Reutlinger, M., Karypis, G., Li, M.: Benchmarking accuracy and generalizability of four graph neural networks using large in vitro ADME datasets from different chemical spaces. Mol. Inform. 41(8), 2100321 (2022). https:\/\/doi.org\/10.1002\/minf.202100321, https:\/\/onlinelibrary.wiley.com\/doi\/abs\/10.1002\/minf.202100321","DOI":"10.1002\/minf.202100321"},{"key":"7_CR5","doi-asserted-by":"publisher","unstructured":"Bule, M., Jalalimanesh, N., Bayrami, Z., Baeeri, M., Abdollahi, M.: The rise of deep learning and transformations in bioactivity prediction power of molecular modeling tools. Chem. Biol. Drug Des. 98(5), 954\u2013967 (2021). https:\/\/doi.org\/10.1111\/cbdd.13750, https:\/\/onlinelibrary.wiley.com\/doi\/abs\/10.1111\/cbdd.13750","DOI":"10.1111\/cbdd.13750"},{"key":"7_CR6","doi-asserted-by":"publisher","unstructured":"Chen, H., Engkvist, O., Wang, Y., Olivecrona, M., Blaschke, T.: The rise of deep learning in drug discovery. Drug Discov. Today 23(6), 1241\u20131250 (2018). https:\/\/doi.org\/10.1016\/j.drudis.2018.01.039, https:\/\/www.sciencedirect.com\/science\/article\/pii\/S1359644617303598","DOI":"10.1016\/j.drudis.2018.01.039"},{"key":"7_CR7","doi-asserted-by":"publisher","unstructured":"Chuang, K.V., Gunsalus, L.M., Keiser, M.J.: Learning molecular representations for medicinal chemistry. J. Med. Chem. 63(16), 8705\u20138722 (2020). https:\/\/doi.org\/10.1021\/acs.jmedchem.0c00385, pMID: 32366098","DOI":"10.1021\/acs.jmedchem.0c00385"},{"key":"7_CR8","doi-asserted-by":"publisher","unstructured":"David Z\u00a0Huang, J.C.B., Bahmanyar, S.S.: The challenges of generalizability in artificial intelligence for ADME\/TOX endpoint and activity prediction. Expert Opin. Drug Discov. 16(9), 1045\u20131056 (2021). https:\/\/doi.org\/10.1080\/17460441.2021.1901685, pMID: 33739897","DOI":"10.1080\/17460441.2021.1901685"},{"key":"7_CR9","unstructured":"Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018)"},{"key":"7_CR10","doi-asserted-by":"publisher","unstructured":"Ektefaie, Y., Shen, A., Bykova, D., Marin, M., Zitnik, M., Farhat, M.: Evaluating generalizability of artificial intelligence models for molecular datasets. bioRxiv (2024). https:\/\/doi.org\/10.1101\/2024.02.25.581982, https:\/\/www.biorxiv.org\/content\/early\/2024\/02\/28\/2024.02.25.581982","DOI":"10.1101\/2024.02.25.581982"},{"key":"7_CR11","unstructured":"Fabian, B., et al.: Molecular representation learning with language models and domain-relevant auxiliary tasks. In: Proceedings of the NeurIPS 2020 Workshop on Machine Learning for Molecules (2020)"},{"key":"7_CR12","doi-asserted-by":"publisher","unstructured":"Glavatsk\u00edkh, M., Leguy, J., Hunault, G., Cauchy, T., Da\u00a0Mota, B.: Dataset\u2019s chemical diversity limits the generalizability of machine learning predictions. J. Cheminform. 11(1), 69 (2019). https:\/\/doi.org\/10.1186\/s13321-019-0391-2","DOI":"10.1186\/s13321-019-0391-2"},{"issue":"6","key":"7_CR13","doi-asserted-by":"publisher","first-page":"2198","DOI":"10.1039\/d0sc04823b","volume":"12","author":"Y Guan","year":"2021","unstructured":"Guan, Y., et al.: Regio-selectivity prediction with a machine-learned reaction representation and on-the-fly quantum mechanical descriptors. Chem. Sci. 12(6), 2198\u20132208 (2021). https:\/\/doi.org\/10.1039\/d0sc04823b","journal-title":"Chem. Sci."},{"key":"7_CR14","doi-asserted-by":"publisher","unstructured":"Hoja, J., et al.: Qm7-x, a comprehensive dataset of quantum-mechanical properties spanning the chemical space of small organic molecules. Sci. Data 8(1), 43 (2021). https:\/\/doi.org\/10.1038\/s41597-021-00812-2","DOI":"10.1038\/s41597-021-00812-2"},{"key":"7_CR15","unstructured":"Hu, W., et al.: Strategies for pre-training graph neural networks. In: International Conference on Learning Representations (2020). https:\/\/openreview.net\/forum?id=HJlWWJSFDH"},{"issue":"10","key":"7_CR16","doi-asserted-by":"publisher","first-page":"1033","DOI":"10.1038\/s41589-022-01131-2","volume":"18","author":"K Huang","year":"2022","unstructured":"Huang, K., et al.: Artificial intelligence foundation for therapeutic science. Nat. Chem. Biol. 18(10), 1033\u20131036 (2022). https:\/\/doi.org\/10.1038\/s41589-022-01131-2","journal-title":"Nat. Chem. Biol."},{"key":"7_CR17","doi-asserted-by":"publisher","unstructured":"Isert, C., Atz, K., Jim\u00e9nez-Luna, J., Schneider, G.: QMugs, quantum mechanical properties of drug-like molecules. Sci. Data 9(1) (2022). https:\/\/doi.org\/10.1038\/s41597-022-01390-7","DOI":"10.1038\/s41597-022-01390-7"},{"key":"7_CR18","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1038\/d41573-022-00025-1","volume":"21","author":"MK Jayatunga","year":"2022","unstructured":"Jayatunga, M.K., Xie, W., Ruder, L., Schulze, U., Meier, C.: Ai in small-molecule drug discovery: a coming wave. Nat. Rev. Drug Discov. 21, 175\u2013176 (2022)","journal-title":"Nat. Rev. Drug Discov."},{"key":"7_CR19","doi-asserted-by":"publisher","unstructured":"Kaufman, B., et al.: COATI: multimodal contrastive pretraining for representing and traversing chemical space. J. Chem. Inform. Model. 64(4), 1145\u20131157 (2024). https:\/\/doi.org\/10.1021\/acs.jcim.3c01753, pMID: 38316665","DOI":"10.1021\/acs.jcim.3c01753"},{"key":"7_CR20","doi-asserted-by":"publisher","unstructured":"Li, M.M., Huang, K., Zitnik, M.: Graph representation learning in biomedicine and healthcare. Nat. Biomed. Eng. 6(12), 1353\u20131369 (2022). https:\/\/doi.org\/10.1038\/s41551-022-00942-x","DOI":"10.1038\/s41551-022-00942-x"},{"key":"7_CR21","doi-asserted-by":"crossref","unstructured":"Medrano Sandonas, L., et al.: Dataset for quantum-mechanical exploration of conformers and solvent effects in large drug-like molecules. Sci. Data 11(1), 742 (2024)","DOI":"10.1038\/s41597-024-03521-8"},{"key":"7_CR22","unstructured":"M\u00fcller, L., Galkin, M., Morris, C., Ramp\u00e1\u0161ek, L.: Attending to graph transformers. Transactions on Machine Learning Research (2024). https:\/\/openreview.net\/forum?id=HhbqHBBrfZ"},{"issue":"6","key":"7_CR23","doi-asserted-by":"publisher","first-page":"1300","DOI":"10.1021\/acs.jcim.7b00083","volume":"57","author":"M Nakata","year":"2017","unstructured":"Nakata, M., Shimazaki, T.: PubChemQC project: a large-scale first-principles electronic structure database for data-driven chemistry. J. Chem. Inf. Model. 57(6), 1300\u20131308 (2017). https:\/\/doi.org\/10.1021\/acs.jcim.7b00083","journal-title":"J. Chem. Inf. Model."},{"key":"7_CR24","doi-asserted-by":"publisher","unstructured":"Nugmanov, R., Dyubankova, N., Gedich, A., Wegner, J.K.: Bidirectional graphormer for reactivity understanding: neural network trained to reaction atom-to-atom mapping task. J. Chem. Inform. Model. 62(14), 3307\u20133315 (2022). https:\/\/doi.org\/10.1021\/acs.jcim.2c00344, pMID: 35792579","DOI":"10.1021\/acs.jcim.2c00344"},{"key":"7_CR25","doi-asserted-by":"publisher","unstructured":"Wang, Y., Xu, C., Li, Z., Barati\u00a0Farimani, A.: Denoise pretraining on nonequilibrium molecules for accurate and transferable neural potentials. J. Chem. Theory Comput. 19(15), 5077\u20135087 (2023). https:\/\/doi.org\/10.1021\/acs.jctc.3c00289, pMID: 37390120","DOI":"10.1021\/acs.jctc.3c00289"},{"key":"7_CR26","doi-asserted-by":"crossref","unstructured":"Xia, J., et al.: Mole-BERT: rethinking pre-training graph neural networks for molecules. In: The Eleventh International Conference on Learning Representations (2023). https:\/\/openreview.net\/forum?id=jevY-DtiZTR","DOI":"10.26434\/chemrxiv-2023-dngg4"},{"key":"7_CR27","doi-asserted-by":"publisher","unstructured":"Xia, J., Zhu, Y., Du, Y., Li, S.Z.: A systematic survey of chemical pre-trained models. In: Elkind, E. (ed.) Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23. pp. 6787\u20136795. International Joint Conferences on Artificial Intelligence Organization (2023). https:\/\/doi.org\/10.24963\/ijcai.2023\/760, survey Track","DOI":"10.24963\/ijcai.2023\/760"},{"key":"7_CR28","unstructured":"Ying, C., et al.: Do transformers really perform badly for graph representation? In: Advances in Neural Information Processing Systems, vol.\u00a034, pp. 28877\u201328888. Curran Associates, Inc. (2021). https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2021\/file\/f1c1592588411002af340cbaedd6fc33-Paper.pdf"}],"container-title":["Lecture Notes in Computer Science","AI in Drug Discovery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-72381-0_7","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,19]],"date-time":"2024-09-19T13:11:43Z","timestamp":1726751503000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-72381-0_7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,20]]},"ISBN":["9783031723803","9783031723810"],"references-count":28,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-72381-0_7","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"value":"0302-9743","type":"print"},{"value":"1611-3349","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,20]]},"assertion":[{"value":"20 September 2024","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"The authors have no competing interests to declare that\u00a0are relevant to the content of this article.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Disclosure of Interests"}},{"value":"AIDD","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Workshop on AI in Drug Discovery","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Lugano","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Switzerland","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2024","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"19 September 2024","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"19 September 2024","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"1","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"aidd2024","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/e-nns.org\/wp-content\/uploads\/2024\/ICANN2024-AIDrugDis-CfP_final.pdf","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}