{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T06:15:20Z","timestamp":1784614520092,"version":"3.55.0"},"reference-count":48,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2021,5,5]],"date-time":"2021-05-05T00:00:00Z","timestamp":1620172800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/100007219","name":"Shanghai Municipal Natural Science Foundation","doi-asserted-by":"publisher","award":["kq2014144"],"award-info":[{"award-number":["kq2014144"]}],"id":[{"id":"10.13039\/100007219","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100007131","name":"Changzhou Science and Technology Bureau","doi-asserted-by":"publisher","award":["kq2001034"],"award-info":[{"award-number":["kq2001034"]}],"id":[{"id":"10.13039\/501100007131","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100007225","name":"Ministry of Science and Technology","doi-asserted-by":"publisher","award":["2018YFB1003203"],"award-info":[{"award-number":["2018YFB1003203"]}],"id":[{"id":"10.13039\/100007225","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Science Foundation of China","doi-asserted-by":"publisher","award":["U1811462"],"award-info":[{"award-number":["U1811462"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,11,5]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Motivation: Accurate and efficient prediction of molecular properties is one of the fundamental issues in drug design and discovery pipelines. Traditional feature engineering-based approaches require extensive expertise in the feature design and selection process. With the development of artificial intelligence (AI) technologies, data-driven methods exhibit unparalleled advantages over the feature engineering-based methods in various domains. Nevertheless, when applied to molecular property prediction, AI models usually suffer from the scarcity of labeled data and show poor generalization ability.<\/jats:p>\n               <jats:p>Results: In this study, we proposed molecular graph BERT (MG-BERT), which integrates the local message passing mechanism of graph neural networks (GNNs) into the powerful BERT model to facilitate learning from molecular graphs. Furthermore, an effective self-supervised learning strategy named masked atoms prediction was proposed to pretrain the MG-BERT model on a large amount of unlabeled data to mine context information in molecules. We found the MG-BERT model can generate context-sensitive atomic representations after pretraining and transfer the learned knowledge to the prediction of a variety of molecular properties. The experimental results show that the pretrained MG-BERT model with a little extra fine-tuning can consistently outperform the state-of-the-art methods on all 11 ADMET datasets. Moreover, the MG-BERT model leverages attention mechanisms to focus on atomic features essential to the target property, providing excellent interpretability for the trained model. The MG-BERT model does not require any hand-crafted feature as input and is more reliable due to its excellent interpretability, providing a novel framework to develop state-of-the-art models for a wide range of drug discovery tasks.<\/jats:p>","DOI":"10.1093\/bib\/bbab152","type":"journal-article","created":{"date-parts":[[2021,4,1]],"date-time":"2021-04-01T11:12:58Z","timestamp":1617275578000},"source":"Crossref","is-referenced-by-count":147,"title":["MG-BERT: leveraging unsupervised atomic representation learning for molecular property prediction"],"prefix":"10.1093","volume":"22","author":[{"given":"Xiao-Chen","family":"Zhang","sequence":"first","affiliation":[{"name":"State Key Laboratory of High-Performance Computing, School of Computer Science, National University of Defense Technology, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cheng-Kun","family":"Wu","sequence":"additional","affiliation":[{"name":"State Key Laboratory of High-Performance Computing, School of Computer Science, National University of Defense Technology, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhi-Jiang","family":"Yang","sequence":"additional","affiliation":[{"name":"Xiangya School of Pharmaceutical Sciences, Central South University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhen-Xing","family":"Wu","sequence":"additional","affiliation":[{"name":"College of Pharmaceutical Sciences, Zhengjiang University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jia-Cai","family":"Yi","sequence":"additional","affiliation":[{"name":"State Key Laboratory of High-Performance Computing, School of Computer Science, National University of Defense Technology, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chang-Yu","family":"Hsieh","sequence":"additional","affiliation":[{"name":"Tencent Quantum Laboratory since 2018. He received his PhD degree in Physics from the University of Ottawa in 2012 and worked as a postdoctoral researcher at the University of Toronto (2012\u20132013) and Massachusetts Institute of Technology (2013\u20132016), respectively. Before joining Tencent, he worked as a senior researcher at Singapore-MIT Alliance for Science and Technology (2017\u20132018)"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ting-Jun","family":"Hou","sequence":"additional","affiliation":[{"name":"College of Pharmaceutical Sciences, Zhejiang University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dong-Sheng","family":"Cao","sequence":"additional","affiliation":[{"name":"Xiangya School of Pharmaceutical Sciences, Central South University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2021,5,5]]},"reference":[{"key":"2021110814301117600_ref1","first-page":"279","article-title":"Drug design and discovery: principles and applications","volume-title":"Molecules","author":"Zhou","year":"2017"},{"key":"2021110814301117600_ref2","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1146\/annurev.pa.27.040187.001205","article-title":"Computer-aided drug design","volume":"27","author":"Marshall","year":"1987","journal-title":"Annu Rev Pharmacol"},{"key":"2021110814301117600_ref3","doi-asserted-by":"crossref","first-page":"33","DOI":"10.2174\/1568005033342145","article-title":"Strategy of computer-aided drug design","volume":"3","author":"Veselovsky","year":"2003","journal-title":"Current Drug Targets-Infectious Disorders"},{"key":"2021110814301117600_ref4","doi-asserted-by":"crossref","first-page":"579","DOI":"10.1093\/bib\/bbp023","article-title":"Recent advances in computer-aided drug design","volume":"10","author":"Song","year":"2009","journal-title":"Brief Bioinform"},{"key":"2021110814301117600_ref5","first-page":"25","article-title":"Machine learning: an indispensable tool in bioinformatics","volume-title":"Methods Mol Biol","author":"Inza","year":"2010"},{"key":"2021110814301117600_ref6","doi-asserted-by":"crossref","first-page":"375","DOI":"10.1016\/0278-6915(90)90112-Z","article-title":"Survey of the QSAR and in vitro approaches for developing non-animal methods to supersede the in vivo LD50 test","volume":"28","author":"Phillips","year":"1990","journal-title":"Food Chem Toxicol"},{"issue":"2","key":"2021110814301117600_ref7","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1021\/ci990162i","article-title":"The characterization of chemical structures using molecular properties","volume":"40","author":"Livingstone","year":"2000","journal-title":"A Survey, J Chem Inf Comput Sci"},{"key":"2021110814301117600_ref8","doi-asserted-by":"crossref","first-page":"742","DOI":"10.1021\/ci100050t","article-title":"Extended-connectivity fingerprints","volume":"50","author":"Rogers","year":"2010","journal-title":"J Chem Inf Model"},{"key":"2021110814301117600_ref9","first-page":"bbaa092","article-title":"Different molecular enumeration influences in deep learning: an example using aqueous solubility","author":"Chen","year":"2020","journal-title":"Brief Bioinform"},{"key":"2021110814301117600_ref10","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1007\/978-1-4020-9783-6_3","article-title":"Molecular descriptors","volume-title":"Recent advances in QSAR studies, Methods and applications","author":"Consonni","year":"2010"},{"key":"2021110814301117600_ref11","volume-title":"Handbook of Molecular Descriptors","author":"Todeschini","year":"2002"},{"key":"2021110814301117600_ref12","first-page":"2818","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition","author":"Szegedy","year":"2016"},{"key":"2021110814301117600_ref13","first-page":"630","volume-title":"European Conference on Computer Vision","author":"He","year":"2016"},{"key":"2021110814301117600_ref14","article-title":"Attention is all you need","author":"Vaswani","year":"2017","journal-title":"arXiv preprint"},{"key":"2021110814301117600_ref15","article-title":"Bert: pre-training of deep bidirectional transformers for language understanding","author":"Devlin","year":"2018","journal-title":"arXiv"},{"key":"2021110814301117600_ref16","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1038\/nature24270","article-title":"Mastering the game of go without human knowledge","volume":"550","author":"Silver","year":"2017","journal-title":"Nature"},{"key":"2021110814301117600_ref17","article-title":"SMILES enumeration as data augmentation for neural network modeling of molecules","author":"Bjerrum","year":"2017","journal-title":"arXiv"},{"key":"2021110814301117600_ref18","article-title":"Neural message passing for quantum chemistry","author":"Gilmer","journal-title":"Proceedings of the 34th International Conference on Machine Learning, Sydney, NSW, Australia, 2017. p. 1263\u20131272"},{"key":"2021110814301117600_ref19","doi-asserted-by":"crossref","first-page":"1692","DOI":"10.1039\/C8SC04175J","article-title":"Learning continuous and data-driven molecular descriptors by translating equivalent chemical representations","volume":"10","author":"Winter","year":"2019","journal-title":"Chem Sci"},{"key":"2021110814301117600_ref20","doi-asserted-by":"crossref","first-page":"1520","DOI":"10.1021\/acscentsci.8b00507","article-title":"Potential net for molecular property prediction","volume":"4","author":"Feinberg","year":"2018","journal-title":"ACS Central Science"},{"key":"2021110814301117600_ref21","article-title":"Atomic convolutional networks for predicting protein-ligand binding affinity","author":"Gomes","year":"2017","journal-title":"arXiv"},{"key":"2021110814301117600_ref22","doi-asserted-by":"crossref","first-page":"595","DOI":"10.1007\/s10822-016-9938-8","article-title":"Molecular graph convolutions: moving beyond fingerprints","volume":"30","author":"Kearnes","year":"2016","journal-title":"J Comput Aided Mol Des"},{"issue":"1","key":"2021110814301117600_ref23","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1186\/s13321-020-00423-w","article-title":"Transformer-CNN: Swiss knife for QSAR modeling and interpretation","volume":"12","author":"Karpov","year":"2020","journal-title":"J Cheminform"},{"key":"2021110814301117600_ref24","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1145\/3107411.3107424","volume-title":"Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics","author":"Xu","year":"2017"},{"key":"2021110814301117600_ref25","doi-asserted-by":"crossref","first-page":"3098","DOI":"10.1021\/acs.molpharmaceut.7b00346","article-title":"druGAN: an advanced generative adversarial autoencoder model for de novo generation of new molecules with desired molecular properties in silico","volume":"14","author":"Kadurin","year":"2017","journal-title":"Mol Pharm"},{"key":"2021110814301117600_ref26","doi-asserted-by":"crossref","first-page":"8835","DOI":"10.1021\/acs.jmedchem.9b02187","article-title":"Improvement in ADMET prediction with multitask deep featurization","volume":"63","author":"Feinberg","year":"2020","journal-title":"J Med Chem"},{"key":"2021110814301117600_ref27","article-title":"Graph attention networks","author":"Veli\u010dkovi\u0107"},{"key":"2021110814301117600_ref28","article-title":"Semi-supervised classification with graph convolutional networks","author":"Kipf","journal-title":"International Conference on Learning Representations"},{"issue":"16","key":"2021110814301117600_ref29","doi-asserted-by":"crossref","first-page":"8749","DOI":"10.1021\/acs.jmedchem.9b00959","article-title":"Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism","volume":"63","author":"Xiong","year":"2019","journal-title":"J Med Chem"},{"key":"2021110814301117600_ref30","first-page":"23766","article-title":"Accurate predictions of aqueous solubility of drug molecules via the multilevel graph convolutional network (MGCN) and SchNet architectures","volume":"22","author":"Gao","year":"2020","journal-title":"Journal of Machine Learning Research"},{"key":"2021110814301117600_ref31","article-title":"Edge attention-based multi-relational graph convolutional networks","author":"Shang","year":"2018","journal-title":"arXiv"},{"key":"2021110814301117600_ref32","article-title":"Deepgcns: making gcns go as deep as cnns","author":"Li","year":"2019","journal-title":"arXiv"},{"key":"2021110814301117600_ref33","doi-asserted-by":"crossref","first-page":"146","DOI":"10.1016\/j.inffus.2017.10.006","article-title":"A survey on deep learning for big data","volume":"42","author":"Zhang","year":"2018","journal-title":"Inform Fusion"},{"key":"2021110814301117600_ref34","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1145\/3065386","article-title":"Imagenet classification with deep convolutional neural networks","volume":"60","author":"Krizhevsky","year":"2017","journal-title":"Commun ACM"},{"key":"2021110814301117600_ref35","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1186\/s13321-018-0283-x","article-title":"ADMETlab: a platform for systematic ADMET evaluation based on a comprehensively collected ADMET database","volume":"10","author":"Dong","year":"2018","journal-title":"J Chem"},{"key":"2021110814301117600_ref36","article-title":"A simple framework for contrastive learning of visual representations","author":"Chen","year":"2020","journal-title":"Proceedings of the 37th International Conference on Machine Learning, Virtual Event"},{"key":"2021110814301117600_ref37","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1145\/3307339.3342186","volume-title":"Proceedings of the 10th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics","author":"Wang","year":"2019"},{"key":"2021110814301117600_ref38","doi-asserted-by":"crossref","first-page":"D1100","DOI":"10.1093\/nar\/gkr777","article-title":"ChEMBL: a large-scale bioactivity database for drug discovery","volume":"40","author":"Gaulton","year":"2012","journal-title":"Nucleic Acids Res"},{"key":"2021110814301117600_ref39","doi-asserted-by":"crossref","first-page":"513","DOI":"10.1039\/C7SC02664A","article-title":"MoleculeNet: a benchmark for molecular machine learning","volume":"9","author":"Wu","year":"2018","journal-title":"Chem Sci"},{"key":"2021110814301117600_ref40","article-title":"Relational inductive biases, deep learning, and graph networks","author":"Battaglia","year":"2018","journal-title":"arXiv"},{"key":"2021110814301117600_ref41","article-title":"RDKit: Open-Source Cheminformatics Software","author":"Landrum"},{"key":"2021110814301117600_ref42","article-title":"Adam: a method for stochastic optimization","author":"Kingma","journal-title":"International Conference on Learning Representations"},{"key":"2021110814301117600_ref43","first-page":"1929","article-title":"Dropout: a simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"The Journal of Machine Learning Research"},{"key":"2021110814301117600_ref44","article-title":"Roberta: A robustly optimized bert pretraining approach","author":"Liu","journal-title":"International Conference on Learning Representations"},{"key":"2021110814301117600_ref45","doi-asserted-by":"crossref","first-page":"785","DOI":"10.1145\/2939672.2939785","volume-title":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Chen","year":"2016"},{"issue":"10","key":"2021110814301117600_ref46","doi-asserted-by":"crossref","first-page":"e2","DOI":"10.23915\/distill.00002","article-title":"How to use t-SNE effectively","volume":"1","author":"Wattenberg","year":"2016","journal-title":"Distill"},{"issue":"11","key":"2021110814301117600_ref47","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"Van der Maaten","year":"2008","journal-title":"Journal of Machine Learning Research"},{"key":"2021110814301117600_ref48","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1515\/aiht-2016-67-2801","article-title":"Mutagenic and carcinogenic structural alerts and their mechanisms of action","volume":"67","author":"Plo\u0161nik","year":"2016","journal-title":"Arh Hig Rada Toksikol"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/22\/6\/bbab152\/41088150\/bbab152.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/22\/6\/bbab152\/41088150\/bbab152.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,11,8]],"date-time":"2021-11-08T14:32:31Z","timestamp":1636381951000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbab152\/6265201"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,5]]},"references-count":48,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2021,11,5]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbab152","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,11]]},"published":{"date-parts":[[2021,5,5]]},"article-number":"bbab152"}}