{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,3]],"date-time":"2026-09-03T11:13:15Z","timestamp":1788433995588,"version":"build-2803163510"},"reference-count":56,"publisher":"MDPI AG","issue":"15","license":[{"start":{"date-parts":[[2023,7,26]],"date-time":"2023-07-26T00:00:00Z","timestamp":1690329600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"NSF","award":["2231498"],"award-info":[{"award-number":["2231498"]}]},{"name":"NSF","award":["2023"],"award-info":[{"award-number":["2023"]}]},{"name":"UAlbany SAGES","award":["2231498"],"award-info":[{"award-number":["2231498"]}]},{"name":"UAlbany SAGES","award":["2023"],"award-info":[{"award-number":["2023"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IJMS"],"abstract":"<jats:p>In this study, we introduce semi-supervised machine learning models designed to predict molecular properties. Our model employs a two-stage approach, involving pre-training and fine-tuning. Particularly, our model leverages a substantial amount of labeled and unlabeled data consisting of SMILES strings, a text representation system for molecules. During the pre-training stage, our model capitalizes on the Masked Language Model, which is widely used in natural language processing, for learning molecular chemical space representations. During the fine-tuning stage, our model is trained on a smaller labeled dataset to tackle specific downstream tasks, such as classification or regression. Preliminary results indicate that our model demonstrates comparable performance to state-of-the-art models on the chosen downstream tasks from MoleculeNet. Additionally, to reduce the computational overhead, we propose a new approach taking advantage of 3D compound structures for calculating the attention score used in the end-to-end transformer model to predict anti-malaria drug candidates. The results show that using the proposed attention score, our end-to-end model is able to have comparable performance with pre-trained models.<\/jats:p>","DOI":"10.3390\/ijms241511948","type":"journal-article","created":{"date-parts":[[2023,7,27]],"date-time":"2023-07-27T01:27:46Z","timestamp":1690421266000},"page":"11948","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":["Molecular Descriptors Property Prediction Using Transformer-Based Approach"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2299-0466","authenticated-orcid":false,"given":"Tuan","family":"Tran","sequence":"first","affiliation":[{"name":"Department of Computer Science, University at Albany, Albany, NY 12203, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chinwe","family":"Ekenna","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University at Albany, Albany, NY 12203, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,7,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1093\/biostatistics\/kxx069","article-title":"Estimation of clinical trial success rates and related parameters","volume":"20","author":"Wong","year":"2019","journal-title":"Biostatistics"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1291","DOI":"10.1016\/j.drudis.2016.06.013","article-title":"Descriptors and their selection methods in QSAR analysis: Paradigm for drug design","volume":"21","author":"Danishuddin","year":"2016","journal-title":"Drug Discov. Today"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Mswahili, M.E., Lee, M.J., Martin, G.L., Kim, J., Kim, P., Choi, G.J., and Jeong, Y.S. (2021). Cocrystal prediction using machine learning models and descriptors. Appl. Sci., 11.","DOI":"10.3390\/app11031323"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1080\/1062936X.2020.1740890","article-title":"Classification models for predicting the antimalarial activity against Plasmodium falciparum","volume":"31","author":"Liu","year":"2020","journal-title":"SAR QSAR Environ. Res."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Xu, Z., Wang, S., Zhu, F., and Huang, J. (2017, January 20\u201323). Seq2seq fingerprint: An unsupervised deep molecular embedding for drug discovery. Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, Boston, MA, USA.","DOI":"10.1145\/3107411.3107424"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Zhang, X., Wang, S., Zhu, F., Xu, Z., Wang, Y., and Huang, J. (September, January 29). Seq3seq fingerprint: Towards end-to-end semi-supervised deep drug discovery. Proceedings of the 2018 ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, Washington, DC, USA.","DOI":"10.1145\/3233547.3233548"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Tran, T., and Ekenna, C. (2020, January 16\u201319). Protein binding pose prediction via conditional variational autoencoding for plasmodium falciparum. Proceedings of the 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Seoul, Republic of Korea.","DOI":"10.1109\/BIBM49941.2020.9313491"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1021\/ci00057a005","article-title":"SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules","volume":"28","author":"Weininger","year":"1988","journal-title":"J. Chem. Inf. Comput. Sci."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Wang, S., Guo, Y., Wang, Y., Sun, H., and Huang, J. (2019, January 7\u201310). SMILES-BERT: Large scale unsupervised pre-training for molecular property prediction. Proceedings of the 10th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, Niagara Falls, NY, USA.","DOI":"10.1145\/3307339.3342186"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"3715","DOI":"10.1073\/pnas.91.9.3715","article-title":"Parasite virulence and disease patterns in Plasmodium falciparum malaria","volume":"91","author":"Gupta","year":"1994","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"917","DOI":"10.1038\/nm.4381","article-title":"Antimalarial drug resistance: Linking Plasmodium falciparum parasite biology to the clinic","volume":"23","author":"Blasco","year":"2017","journal-title":"Nat. Med."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1737","DOI":"10.3389\/fmicb.2018.01737","article-title":"Discovering putative prion-like proteins in Plasmodium falciparum: A computational and experimental analysis","volume":"9","author":"Iglesias","year":"2018","journal-title":"Front. Microbiol."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1016\/j.molcel.2011.05.013","article-title":"Opposing effects of glutamine and asparagine govern prion formation by intrinsically disordered proteins","volume":"43","author":"Halfmann","year":"2011","journal-title":"Mol. Cell"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1146\/annurev-biochem-061516-045115","article-title":"Protein misfolding, amyloid formation, and human disease: A summary of progress over the last decade","volume":"86","author":"Chiti","year":"2017","journal-title":"Annu. Rev. Biochem."},{"key":"ref_15","unstructured":"Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O., and Dahl, G.E. (2017, January 6\u201311). Neural message passing for quantum chemistry. Proceedings of the International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"20381","DOI":"10.1038\/s41598-019-56773-5","article-title":"Molecular geometry prediction using a deep generative graph neural network","volume":"9","author":"Mansimov","year":"2019","journal-title":"Sci. Rep."},{"key":"ref_17","first-page":"10-48550","article-title":"Graph attention networks","volume":"1050","author":"Velickovic","year":"2017","journal-title":"Stat"},{"key":"ref_18","unstructured":"Shang, C., Liu, Q., Chen, K.S., Sun, J., Lu, J., Yi, J., and Bi, J. (2018). Edge attention-based multi-relational graph convolutional networks. arXiv."},{"key":"ref_19","unstructured":"Hu, W., Liu, B., Gomes, J., Zitnik, M., Liang, P., Pande, V., and Leskovec, J. (2019). Strategies for pre-training graph neural networks. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1758-2946-5-7","article-title":"InChI-the worldwide chemical structure identifier standard","volume":"5","author":"Heller","year":"2013","journal-title":"J. Cheminform."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"100588","DOI":"10.1016\/j.patter.2022.100588","article-title":"SELFIES and the future of molecular string representations","volume":"3","author":"Krenn","year":"2022","journal-title":"Patterns"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13321-020-00430-x","article-title":"Inductive transfer learning for molecular activity prediction: Next-Gen QSAR Models with MolPMoFiT","volume":"12","author":"Li","year":"2020","journal-title":"J. Cheminform."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"4191","DOI":"10.1021\/acs.jcim.9b01212","article-title":"Predicting binding from screening assays with transformer network embeddings","volume":"60","author":"Morris","year":"2020","journal-title":"J. Chem. Inf. Model."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"793","DOI":"10.1109\/TEVC.2022.3144045","article-title":"Automating Genetic Algorithm Mutations for Molecules Using a Masked Language Model","volume":"26","author":"Blanchard","year":"2022","journal-title":"IEEE Trans. Evol. Comput."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"2111","DOI":"10.1021\/acs.jcim.5b00543","article-title":"Get Your Atoms in Order An Open-Source Implementation of a Novel and Robust Molecular Canonicalization Algorithm","volume":"55","author":"Schneider","year":"2015","journal-title":"J. Chem. Inf. Model."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Neglur, G., Grossman, R.L., and Liu, B. (2005, January 20\u201322). Assigning unique keys to chemical compounds for data integration: Some interesting counter examples. Proceedings of the International Workshop on Data Integration in the Life Sciences, San Diego, CA, USA.","DOI":"10.1007\/11530084_13"},{"key":"ref_27","first-page":"6000","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_28","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv."},{"key":"ref_29","unstructured":"Fabian, B., Edlich, T., Gaspar, H., Segler, M., Meyers, J., Fiscato, M., and Ahmed, M. (2020). Molecular representation learning with language models and domain-relevant auxiliary tasks. arXiv."},{"key":"ref_30","unstructured":"Ahmad, W., Simon, E., Chithrananda, S., Grand, G., and Ramsundar, B. (2022). ChemBERTa-2: Towards Chemical Foundation Models. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"5959","DOI":"10.1021\/ja00226a005","article-title":"Comparative molecular field analysis (CoMFA). 1. Effect of shape on binding of steroids to carrier proteins","volume":"110","author":"Cramer","year":"1988","journal-title":"J. Am. Chem. Soc."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"3183","DOI":"10.1021\/jm980697n","article-title":"Robust QSAR models using Bayesian regularized neural networks","volume":"42","author":"Burden","year":"1999","journal-title":"J. Med. Chem."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1016\/j.taap.2014.12.014","article-title":"Predicting chemically-induced skin reactions. Part I: QSAR models of skin sensitization and their application to identify potentially hazardous compounds","volume":"284","author":"Alves","year":"2015","journal-title":"Toxicol. Appl. Pharmacol."},{"key":"ref_34","first-page":"83","article-title":"Making big sense from big data in toxicology by read-across","volume":"33","author":"Hartung","year":"2016","journal-title":"ALTEX-Altern. Anim. Exp."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1291","DOI":"10.1002\/jcc.24764","article-title":"Deep learning for computational chemistry","volume":"38","author":"Goh","year":"2017","journal-title":"J. Comput. Chem."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Neves, B.J., Braga, R.C., Alves, V.M., Lima, M.N., Cassiano, G.C., Muratov, E.N., Costa, F.T., and Andrade, C.H. (2020). Deep Learning-driven research for drug discovery: Tackling Malaria. Plos Comput. Biol., 16.","DOI":"10.1371\/journal.pcbi.1007025"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1096","DOI":"10.3389\/fphar.2018.01096","article-title":"Using machine learning to predict synergistic antimalarial compound combinations with novel structures","volume":"9","author":"Mason","year":"2018","journal-title":"Front. Pharmacol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1021\/acscentsci.7b00512","article-title":"Generating focused molecule libraries for drug discovery with recurrent neural networks","volume":"4","author":"Segler","year":"2018","journal-title":"ACS Cent. Sci."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1526","DOI":"10.3389\/fphar.2019.01526","article-title":"DeepMalaria: Artificial intelligence driven discovery of potent antiplasmodials","volume":"10","author":"Salem","year":"2020","journal-title":"Front. Pharmacol."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Mswahili, M.E., Martin, G.L., Woo, J., Choi, G.J., and Jeong, Y.S. (2021). Antimalarial Drug Predictions Using Molecular Descriptors and Machine Learning against Plasmodium Falciparum. Biomolecules, 11.","DOI":"10.3390\/biom11121750"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"773","DOI":"10.3389\/fchem.2019.00773","article-title":"Integrative multi-kinase approach for the identification of potent antiplasmodial hits","volume":"7","author":"Lima","year":"2019","journal-title":"Front. Chem."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"3370","DOI":"10.1021\/acs.jcim.9b00237","article-title":"Analyzing learned molecular representations for property prediction","volume":"59","author":"Yang","year":"2019","journal-title":"J. Chem. Inf. Model."},{"key":"ref_43","unstructured":"Ho, T.K. (1995, January 14\u201316). Random decision forests. Proceedings of the 3rd International Conference on Document Analysis and Recognition, Montreal, QC, Canada."},{"key":"ref_44","unstructured":"Kipf, T.N., and Welling, M. (2016). Semi-supervised classification with graph convolutional networks. arXiv."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Cho, K., Van Merri\u00ebnboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv.","DOI":"10.3115\/v1\/D14-1179"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Graves, A. (2013). Generating sequences with recurrent neural networks. arXiv.","DOI":"10.1007\/978-3-642-24797-2_3"},{"key":"ref_47","unstructured":"Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv."},{"key":"ref_48","unstructured":"Chithrananda, S., Grand, G., and Ramsundar, B. (2020). ChemBERTa: Large-scale self-supervised pretraining for molecular property prediction. arXiv."},{"key":"ref_49","first-page":"9689","article-title":"Evaluating protein transfer learning with TAPE","volume":"32","author":"Rao","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_50","first-page":"1970","article-title":"Se (3)-transformers: 3d roto-translation equivariant attention networks","volume":"33","author":"Fuchs","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"704","DOI":"10.1038\/s41594-021-00650-1","article-title":"AlphaFold2 and the future of structural biology","volume":"28","author":"Cramer","year":"2021","journal-title":"Nat. Struct. Mol. Biol."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"D1102","DOI":"10.1093\/nar\/gky1033","article-title":"PubChem 2019 update: Improved access to chemical data","volume":"47","author":"Kim","year":"2019","journal-title":"Nucleic Acids Res."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"513","DOI":"10.1039\/C7SC02664A","article-title":"MoleculeNet: A benchmark for molecular machine learning","volume":"9","author":"Wu","year":"2018","journal-title":"Chem. Sci."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., and Funtowicz, M. (2020, January 16\u201320). Transformers: State-of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online.","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"ref_55","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Tran, T., and Ekenna, C. (2022, January 6\u20138). Molecular Descriptors Property Prediction via a Natural Language Processing Approach. Proceedings of the 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Las Vegas, NV, USA.","DOI":"10.1109\/BIBM55620.2022.9995393"}],"container-title":["International Journal of Molecular Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1422-0067\/24\/15\/11948\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:19:10Z","timestamp":1760127550000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1422-0067\/24\/15\/11948"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,26]]},"references-count":56,"journal-issue":{"issue":"15","published-online":{"date-parts":[[2023,8]]}},"alternative-id":["ijms241511948"],"URL":"https:\/\/doi.org\/10.3390\/ijms241511948","relation":{},"ISSN":["1422-0067"],"issn-type":[{"value":"1422-0067","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,26]]}}}