{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:34:20Z","timestamp":1782840860837,"version":"3.54.5"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,12,9]],"date-time":"2024-12-09T00:00:00Z","timestamp":1733702400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,12,9]],"date-time":"2024-12-09T00:00:00Z","timestamp":1733702400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100010661","name":"Horizon 2020 Framework Programme","doi-asserted-by":"publisher","award":["956832"],"award-info":[{"award-number":["956832"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Helmholtz Zentrum M\u00fcnchen - Deutsches Forschungszentrum f\u00fcr Gesundheit und Umwelt (GmbH)"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Cheminform"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Hyperparameter optimization is very frequently employed in machine learning. However, an optimization of a large space of parameters could result in overfitting of models. In recent studies on solubility prediction the authors collected seven thermodynamic and kinetic solubility datasets from different data sources. They used state-of-the-art graph-based methods and compared models developed for each dataset using different data cleaning protocols and hyperparameter optimization. In our study we showed that hyperparameter optimization did not always result in better models, possibly due to overfitting when using the same statistical measures. Similar results could be calculated using pre-set hyperparameters, reducing the computational effort by around 10,000 times. We also extended the previous analysis by adding a representation learning method based on Natural Language Processing of smiles called Transformer CNN. We show that across all analyzed sets using exactly the same protocol, Transformer CNN provided better results than graph-based methods for 26 out of 28 pairwise comparisons by using only a tiny fraction of time as compared to other methods. Last but not least we stressed the importance of comparing calculation results using exactly the same statistical measures.<\/jats:p><jats:p><jats:bold>Scientific Contribution<\/jats:bold> We showed that models with pre-optimized hyperparameters can suffer from overfitting and that using pre-set hyperparameters yields similar performances but four orders faster. Transformer CNN provided significantly higher accuracy compared to other investigated methods.<\/jats:p>","DOI":"10.1186\/s13321-024-00934-w","type":"journal-article","created":{"date-parts":[[2024,12,9]],"date-time":"2024-12-09T16:12:50Z","timestamp":1733760770000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":42,"title":["Be aware of overfitting by hyperparameter optimization!"],"prefix":"10.1186","volume":"16","author":[{"given":"Igor V.","family":"Tetko","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ruud","family":"van Deursen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guillaume","family":"Godin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,12,9]]},"reference":[{"key":"934_CR1","doi-asserted-by":"publisher","first-page":"421","DOI":"10.1136\/bmj.1.328.421","volume":"1","author":"BW Richardson","year":"1867","unstructured":"Richardson BW (1867) Lectures on experimental and practical medicine. Br Med J 1:421\u2013422","journal-title":"Br Med J"},{"key":"934_CR2","doi-asserted-by":"publisher","first-page":"8749","DOI":"10.1021\/acs.jmedchem.9b00959","volume":"63","author":"Z Xiong","year":"2020","unstructured":"Xiong Z et al (2020) Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. J Med Chem 63:8749\u20138760","journal-title":"J Med Chem"},{"key":"934_CR3","doi-asserted-by":"publisher","first-page":"20981","DOI":"10.1039\/C8CP01786G","volume":"20","author":"S Boothroyd","year":"2018","unstructured":"Boothroyd S, Kerridge A, Broo A, Buttar D, Anwar J (2018) Solubility prediction from first principles: a density of states approach. Phys Chem Chem Phys 20:20981\u201320987","journal-title":"Phys Chem Chem Phys"},{"key":"934_CR4","doi-asserted-by":"publisher","DOI":"10.1002\/cem.3349","volume":"35","author":"M Lovri\u0107","year":"2021","unstructured":"Lovri\u0107 M et al (2021) Machine learning in prediction of intrinsic aqueous solubility of drug-like compounds: generalization, complexity, or predictive ability? J Chemom 35:e3349","journal-title":"J Chemom"},{"key":"934_CR5","doi-asserted-by":"publisher","first-page":"769","DOI":"10.1007\/s10822-019-00242-8","volume":"34","author":"Z Xia","year":"2020","unstructured":"Xia Z, Karpov P, Popowicz G, Tetko IV (2020) Focused library generator: case of Mdmx inhibitors. J Comput Aided Mol Des 34:769\u2013782","journal-title":"J Comput Aided Mol Des"},{"key":"934_CR6","doi-asserted-by":"publisher","DOI":"10.1016\/j.isci.2020.101961","volume":"24","author":"MC Sorkun","year":"2021","unstructured":"Sorkun MC, Koelman JMVA, Er S (2021) Pushing the limits of solubility prediction via quality-oriented data selection. iScience 24:101961","journal-title":"iScience"},{"key":"934_CR7","doi-asserted-by":"publisher","DOI":"10.1016\/j.slasd.2024.01.005","volume":"29","author":"A Hunklinger","year":"2024","unstructured":"Hunklinger A, Hartog P, \u0160\u00edcho M, Godin G, Tetko IV (2024) The openOCHEM consensus model is the best-performing open-source predictive model in the First EUOS\/SLAS joint compound solubility challenge. SLAS Discov 29:100144","journal-title":"SLAS Discov"},{"key":"934_CR8","doi-asserted-by":"publisher","first-page":"1488","DOI":"10.1021\/ci000392t","volume":"41","author":"IV Tetko","year":"2001","unstructured":"Tetko IV, Tanchuk VY, Kasheva TN, Villa AE (2001) Estimation of aqueous solubility of chemical compounds using E-state indices. J Chem Inf Comput Sci 41:1488\u20131493","journal-title":"J Chem Inf Comput Sci"},{"key":"934_CR9","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1038\/s41597-019-0151-1","volume":"6","author":"MC Sorkun","year":"2019","unstructured":"Sorkun MC, Khetan A, Er S (2019) AqSolDB, a curated reference set of aqueous solubility and 2D descriptors for a diverse set of compounds. Sci Data 6:143","journal-title":"Sci Data"},{"key":"934_CR10","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1038\/s41597-022-01154-3","volume":"9","author":"J Meng","year":"2022","unstructured":"Meng J et al (2022) Boosting the predictive performance with aqueous solubility dataset curation. Sci Data 9:71","journal-title":"Sci Data"},{"key":"934_CR11","doi-asserted-by":"publisher","first-page":"533","DOI":"10.1007\/s10822-011-9440-2","volume":"25","author":"I Sushko","year":"2011","unstructured":"Sushko I et al (2011) Online chemical modeling environment (OCHEM): web platform for data storage, model development and publishing of chemical information. J Comput Aided Mol Des 25:533\u2013554","journal-title":"J Comput Aided Mol Des"},{"key":"934_CR12","doi-asserted-by":"publisher","first-page":"773","DOI":"10.1021\/ci9901338","volume":"40","author":"J Huuskonen","year":"2000","unstructured":"Huuskonen J (2000) Estimation of aqueous solubility for a diverse set of organic compounds based on molecular topology. J Chem Inf Comput Sci 40:773\u2013777","journal-title":"J Chem Inf Comput Sci"},{"key":"934_CR13","doi-asserted-by":"publisher","first-page":"1000","DOI":"10.1021\/ci034243x","volume":"44","author":"JS Delaney","year":"2004","unstructured":"Delaney JS (2004) ESOL: estimating aqueous solubility directly from molecular structure. J Chem Inf Comput Sci 44:1000\u20131005","journal-title":"J Chem Inf Comput Sci"},{"key":"934_CR14","unstructured":"BCCG-A233-Analiza-Solubility-Assay, Burnham Center for Chemical Genomics. https:\/\/pubchem.ncbi.nlm.nih.gov\/bioassay\/1996."},{"key":"934_CR15","doi-asserted-by":"publisher","first-page":"4127","DOI":"10.1016\/j.bmc.2011.05.005","volume":"19","author":"R Guha","year":"2011","unstructured":"Guha R et al (2011) Exploratory analysis of kinetic solubility measurements of a small molecule library. Bioorg Med Chem 19:4127\u20134134","journal-title":"Bioorg Med Chem"},{"key":"934_CR16","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1021\/ci100364a","volume":"51","author":"T Cheng","year":"2011","unstructured":"Cheng T, Li Q, Wang Y, Bryant SH (2011) Binary classification of aqueous solubility using support vector machines with reduction and recombination feature selection. J Chem Inf Model 51:229\u2013236","journal-title":"J Chem Inf Model"},{"key":"934_CR17","unstructured":"GitHub - Mengjintao\/SolCuration. https:\/\/github.com\/Mengjintao\/SolCuration."},{"key":"934_CR18","doi-asserted-by":"publisher","first-page":"3370","DOI":"10.1021\/acs.jcim.9b00237","volume":"59","author":"K Yang","year":"2019","unstructured":"Yang K et al (2019) Analyzing learned molecular representations for property prediction. J Chem Inf Model 59:3370\u20133388","journal-title":"J Chem Inf Model"},{"key":"934_CR19","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1186\/s13321-020-00423-w","volume":"12","author":"P Karpov","year":"2020","unstructured":"Karpov P, Godin G, Tetko IV (2020) Transformer-CNN: Swiss knife for QSAR modeling and interpretation. J Cheminformatics 12:17","journal-title":"J Cheminformatics"},{"key":"934_CR20","unstructured":"GitHub - aimat-lab\/gcnn_keras: Graph convolutions in Keras with TensorFlow, PyTorch or Jax. https:\/\/github.com\/aimat-lab\/gcnn_keras."},{"key":"934_CR21","doi-asserted-by":"publisher","first-page":"831","DOI":"10.1007\/978-3-030-30493-5_79","volume-title":"Artificial neural networks and machine learning \u2013 ICANN 2019 workshop and special sessions","author":"IV Tetko","year":"2019","unstructured":"Tetko IV, Karpov P, Bruno E, Kimber TB, Godin G (2019) Augmentation is what you need! In: Tetko IV, K\u016frkov\u00e1 V, Karpov P, Theis F (eds) Artificial neural networks and machine learning \u2013 ICANN 2019 workshop and special sessions. Cham, Springer International Publishing, pp 831\u2013835. https:\/\/doi.org\/10.1007\/978-3-030-30493-5_79"},{"key":"934_CR22","unstructured":"GitHub - bigchem\/transformer-cnn: Transformer CNN for QSAR\/QSPR modelling. https:\/\/github.com\/bigchem\/transformer-cnn."},{"key":"934_CR23","unstructured":"openochem \u00b7 GitHub. https:\/\/github.com\/openochem."},{"key":"934_CR24","unstructured":"Vaswani A. et al. Attention Is All You Need. ArXiv170603762 Cs (2017)."},{"key":"934_CR25","unstructured":"Online Chemical Modeling Environment. https:\/\/solub.ochem.eu\/home\/show.do."},{"key":"934_CR26","unstructured":"openochem\/solub at main \u00b7 openochem\/openochem \u00b7 GitHub. https:\/\/github.com\/openochem\/openochem\/tree\/main\/solub."},{"key":"934_CR27","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1002\/minf.201300030","volume":"33","author":"S Vorberg","year":"2014","unstructured":"Vorberg S, Tetko IV (2014) Modeling the biodegradability of chemical compounds using the online CHEmical Modeling Environment (OCHEM). Mol Inform 33:73\u201385","journal-title":"Mol Inform"},{"key":"934_CR28","doi-asserted-by":"publisher","first-page":"8952","DOI":"10.3390\/su13168952","volume":"13","author":"T Yigitcanlar","year":"2021","unstructured":"Yigitcanlar T, Mehmood R, Corchado JM (2021) Green artificial intelligence: towards an efficient, sustainable and equitable technology for smart cities and futures. Sustainability 13:8952","journal-title":"Sustainability"},{"key":"934_CR29","doi-asserted-by":"publisher","first-page":"826","DOI":"10.1021\/ci00027a006","volume":"35","author":"IV Tetko","year":"1995","unstructured":"Tetko IV, Livingstone DJ, Luik AI (1995) Neural network studies. 1. Comparison of overfitting and overtraining. J Chem Inf Comput Sci 35:826\u2013833","journal-title":"J Chem Inf Comput Sci"},{"key":"934_CR30","doi-asserted-by":"publisher","first-page":"1269","DOI":"10.1613\/jair.1.13643","volume":"74","author":"AI Cowen-Rivers","year":"2022","unstructured":"Cowen-Rivers AI et al (2022) HEBO: pushing the limits of sample-efficient hyper-parameter optimisation. J Artif Intell Res 74:1269\u20131349","journal-title":"J Artif Intell Res"},{"key":"934_CR31","unstructured":"Griffiths R.-R et al. (2022) GAUCHE: a library for Gaussian processes in chemistry."},{"key":"934_CR32","unstructured":"Moss HB, Beck D, Gonzalez J, Leslie DS, Rayson P (2020) BOSS: Bayesian optimization over string spaces. ArXiv E-Prints arXiv:2010.00979"},{"key":"934_CR33","unstructured":"Moss HB, Griffiths R-R (2020) Gaussian process molecule property prediction with FlowMO. ArXiv E-Prints arXiv:2010.01118"},{"key":"934_CR34","doi-asserted-by":"publisher","first-page":"1733","DOI":"10.1021\/ci800151m","volume":"48","author":"IV Tetko","year":"2008","unstructured":"Tetko IV et al (2008) Critical assessment of QSAR models of environmental toxicity against Tetrahymena pyriformis: focusing on applicability domain and overfitting by variable selection. J Chem Inf Model 48:1733\u20131746","journal-title":"J Chem Inf Model"},{"key":"934_CR35","doi-asserted-by":"publisher","DOI":"10.1016\/j.slasd.2024.100155","volume":"29","author":"W Wang","year":"2024","unstructured":"Wang W, Tang J, Zaliani A (2024) Outline and background for the EU-OS solubility prediction challenge. SLAS Discov. 29:100155","journal-title":"SLAS Discov."},{"key":"934_CR36","unstructured":"EPA ToxCast LELPredictor Marathon Match Results Summary. (2015) https:\/\/web.archive.org\/web\/20150416015853\/http:\/\/www.epa.gov\/ncct\/download_files\/ToxCastMMResultSummary.pdf"},{"key":"934_CR37","doi-asserted-by":"publisher","first-page":"768","DOI":"10.1021\/acs.chemrestox.5b00481","volume":"29","author":"S Novotarskyi","year":"2016","unstructured":"Novotarskyi S et al (2016) ToxCast EPA in vitro to in vivo challenge: insight into the rank-I model. Chem Res Toxicol 29:768\u2013775","journal-title":"Chem Res Toxicol"},{"key":"934_CR38","doi-asserted-by":"publisher","first-page":"825","DOI":"10.1021\/acs.chemrestox.4c00192","volume":"37","author":"IV Tetko","year":"2024","unstructured":"Tetko IV (2024) Tox24 challenge. Chem Res Toxicol 37:825\u2013826","journal-title":"Chem Res Toxicol"},{"key":"934_CR39","doi-asserted-by":"publisher","first-page":"563","DOI":"10.3390\/ijms22020563","volume":"22","author":"IV Semenyuta","year":"2021","unstructured":"Semenyuta IV et al (2021) Structure-activity relationship modeling and experimental validation of the imidazolium and pyridinium based ionic liquids as potential antibacterials of MDR Acinetobacter baumannii and Staphylococcus aureus. Int J Mol Sci 22:563","journal-title":"Int J Mol Sci"},{"key":"934_CR40","doi-asserted-by":"publisher","unstructured":"Standardized Human Olfactory Thresholds. (1990) Oxford University Press. https:\/\/doi.org\/10.1093\/oso\/9780199631469.002.0001.","DOI":"10.1093\/oso\/9780199631469.002.0001"}],"container-title":["Journal of Cheminformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13321-024-00934-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13321-024-00934-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13321-024-00934-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,9]],"date-time":"2024-12-09T17:02:33Z","timestamp":1733763753000},"score":1,"resource":{"primary":{"URL":"https:\/\/jcheminf.biomedcentral.com\/articles\/10.1186\/s13321-024-00934-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,9]]},"references-count":40,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["934"],"URL":"https:\/\/doi.org\/10.1186\/s13321-024-00934-w","relation":{},"ISSN":["1758-2946"],"issn-type":[{"value":"1758-2946","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,9]]},"assertion":[{"value":"23 July 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 November 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 December 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"139"}}