{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T10:26:41Z","timestamp":1787048801201,"version":"build-2736575974"},"reference-count":97,"publisher":"American Chemical Society (ACS)","issue":"10","license":[{"start":{"date-parts":[[2020,8,31]],"date-time":"2020-08-31T00:00:00Z","timestamp":1598832000000},"content-version":"stm-asf","delay-in-days":0,"URL":"https:\/\/doi.org\/10.15223\/policy-029"},{"start":{"date-parts":[[2020,8,31]],"date-time":"2020-08-31T00:00:00Z","timestamp":1598832000000},"content-version":"stm-asf","delay-in-days":0,"URL":"https:\/\/doi.org\/10.15223\/policy-037"},{"start":{"date-parts":[[2020,8,31]],"date-time":"2020-08-31T00:00:00Z","timestamp":1598832000000},"content-version":"stm-asf","delay-in-days":0,"URL":"https:\/\/doi.org\/10.15223\/policy-045"}],"funder":[{"DOI":"10.13039\/100004325","name":"AstraZeneca","doi-asserted-by":"publisher","award":["RG75821"],"award-info":[{"award-number":["RG75821"]}],"id":[{"id":"10.13039\/100004325","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000268","name":"Biotechnology and Biological Sciences Research Council","doi-asserted-by":"publisher","award":["BB\/K011804\/1"],"award-info":[{"award-number":["BB\/K011804\/1"]}],"id":[{"id":"10.13039\/501100000268","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,10,26]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>In the context of bioactivity prediction, the question of how to calibrate a score produced by a machine learning method into a probability of binding to a protein target is not yet satisfactorily addressed. In this study, we compared the performance of three such methods, namely, Platt scaling (PS), isotonic regression (IR), and Venn\u2013ABERS predictors (VA), in calibrating prediction scores obtained from ligand\u2013target prediction comprising the Nai\u0308ve Bayes, support vector machines, and random forest (RF) algorithms. Calibration quality was assessed on bioactivity data available at AstraZeneca for 40 million data points (compound\u2013target pairs) across 2112 targets and performance was assessed using stratified shuffle split (SSS) and leave 20% of scaffolds out (L20SO) validation. VA achieved the best calibration performances across all machine learning algorithms and cross validation methods tested and also the lowest (best) Brier score loss (mean squared difference between the outputted probability estimates assigned to a compound and the actual outcome). In comparison, the PS and IR methods can actually degrade the assigned probability estimates, particularly for the RF for SSS and during L20SO. Sphere exclusion, a method to sample additional (putative) inactive compounds, was shown to inflate the overall Brier score loss performance, through the artificial requirement for inactive molecules to be dissimilar to active compounds, but was shown to result in overconfident estimators. VA was able to successfully calibrate the probability estimates for even small calibration sets. The multiprobability values (lower and upper probability boundary intervals) were shown to produce large discordance for test set molecules that are neither very similar nor very dissimilar to the active training set, which were hence difficult to predict, suggesting that multiprobability discordance can be used as an estimate for target prediction uncertainty. Overall, we were able to show in this work that VA scaling of target prediction models is able to improve probability estimates in all testing instances and is currently being applied for in-house approaches.<\/jats:p>","DOI":"10.1021\/acs.jcim.0c00476","type":"journal-article","created":{"date-parts":[[2020,8,31]],"date-time":"2020-08-31T11:04:48Z","timestamp":1598871888000},"page":"4546-4559","source":"Crossref","is-referenced-by-count":22,"title":["Comparison of Scaling Methods to Obtain Calibrated\nProbabilities of Activity for Protein\u2013Ligand Predictions"],"prefix":"10.1021","volume":"60","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7271-0824","authenticated-orcid":true,"given":"Lewis\nH.","family":"Mervin","sequence":"first","affiliation":[{"name":"AstraZeneca , , ,","place":["Cambridge, U.K., CB2 0AA"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Avid M.","family":"Afzal","sequence":"additional","affiliation":[{"name":"AstraZeneca , , ,","place":["Cambridge, U.K., CB2 0AA"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4970-6461","authenticated-orcid":true,"given":"Ola","family":"Engkvist","sequence":"additional","affiliation":[{"name":"AstraZeneca , , ,","place":["Mo\u0308lndal, Sweden, SE-431 83"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6683-7546","authenticated-orcid":true,"given":"Andreas","family":"Bender","sequence":"additional","affiliation":[{"name":"University of Cambridge , , ,","place":["Cambridge, U.K., CB2 1TN"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"316","published-online":{"date-parts":[[2020,8,31]]},"reference":[{"key":"2026081804380887500_cit1","doi-asserted-by":"publisher","first-page":"1124","DOI":"10.1021\/ci060003g","article-title":"Prediction\nof Biological Targets for Compounds Using\nMultiple-Category Bayesian Models Trained on Chemogenomics Databases","volume":"46","author":"Nidhi","year":"2006","journal-title":"J. Chem. Inf. Model."},{"key":"2026081804380887500_cit2","doi-asserted-by":"publisher","first-page":"5441","DOI":"10.1039\/c8sc00148k","article-title":"Large-Scale\nComparison of Machine Learning Methods for Drug Target Prediction\non Chembl","volume":"9","author":"Mayr","year":"2018","journal-title":"Chem. Sci."},{"key":"2026081804380887500_cit3","unstructured":"Unterthiner, T.; Mayr, A.; Klambauer, G.; Steijaert, M.; Wegner, J. K.; Ceulemans, H.; Hochreiter, S.\n          Deep Learning\nfor Drug Target Prediction. Conference Neural\nInformation Processing Systems Foundation (NIPS 2014), Workshop on\nRepresentation and Learning Methods for Complex Outputs, 2014, 2014."},{"key":"2026081804380887500_cit4","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1186\/s13321-017-0232-0","article-title":"Beyond\nthe Hype: Deep Neural Networks Outperform Established Methods Using\na Chembl Bioactivity Benchmark Set","volume":"9","author":"Lenselink","year":"2017","journal-title":"J. Cheminf."},{"key":"2026081804380887500_cit5","doi-asserted-by":"publisher","first-page":"1593","DOI":"10.1021\/acschembio.7b00209","article-title":"Polypharmacological\nin Silico Bioactivity Profiling\nand Experimental Validation Uncovers Sedative-Hypnotic Effects of\nApproved and Experimental Drugs in Rat","volume":"12","author":"Drakakis","year":"2017","journal-title":"ACS\nChem. Biol."},{"key":"2026081804380887500_cit6","doi-asserted-by":"publisher","first-page":"1401","DOI":"10.1021\/acs.jproteome.6b00618","article-title":"Deep-Learning-Based\nDrug-Target Interaction Prediction","volume":"16","author":"Wen","year":"2017","journal-title":"J. Proteome\nRes."},{"key":"2026081804380887500_cit7","doi-asserted-by":"publisher","first-page":"288","DOI":"10.1016\/j.drudis.2015.12.007","article-title":"In Silico Methods to\nAddress Polypharmacology: Current\nStatus, Applications and Future Perspectives","volume":"21","author":"Lavecchia","year":"2016","journal-title":"Drug Discov. Today"},{"key":"2026081804380887500_cit8","doi-asserted-by":"publisher","first-page":"W357","DOI":"10.1093\/nar\/gkz382","article-title":"Swisstargetprediction: Updated Data\nand New Features for Efficient Prediction of Protein Targets of Small\nMolecules","volume":"47","author":"Daina","year":"2019","journal-title":"Nucleic Acids Res."},{"key":"2026081804380887500_cit9","doi-asserted-by":"publisher","first-page":"586","DOI":"10.1021\/acs.jcim.8b00698","article-title":"A New Approach for Drug Target and Bioactivity Prediction:\nThe Multifingerprint Similarity Search Algorithm (Mussel)","volume":"59","author":"Alberga","year":"2019","journal-title":"J. Chem. Inf. Model."},{"key":"2026081804380887500_cit10","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1021\/acs.jcim.8b00524","article-title":"Polypharmacology\nBrowser Ppb2: Target Prediction Combining\nNearest Neighbors with Machine Learning","volume":"59","author":"Awale","year":"2019","journal-title":"J.\nChem. Inf. Model."},{"key":"2026081804380887500_cit11","doi-asserted-by":"publisher","first-page":"1595","DOI":"10.1021\/ci4002712","article-title":"Comparison of Confirmed\nInactive and Randomly Selected\nCompounds as Negative Training Examples in Support Vector Machine-Based\nVirtual Screening","volume":"53","author":"Heikamp","year":"2013","journal-title":"J. Chem. Inf. Model."},{"key":"2026081804380887500_cit12","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1186\/s13321-015-0098-y","article-title":"Target Prediction Utilising Negative Bioactivity Data\nCovering Large Chemical Space","volume":"7","author":"Mervin","year":"2015","journal-title":"J. Cheminf."},{"key":"2026081804380887500_cit13","doi-asserted-by":"publisher","first-page":"4450","DOI":"10.1021\/acs.jcim.9b00375","article-title":"All-Assay-Max2 Pqsar: Activity Predictions\nas Accurate as Four-Concentration\nIc50s for 8558 Novartis Assays","volume":"59","author":"Martin","year":"2019","journal-title":"J. Chem. Inf.\nModel."},{"key":"2026081804380887500_cit14","doi-asserted-by":"publisher","first-page":"474","DOI":"10.1021\/acs.jmedchem.6b01611","article-title":"Profiling\nPrediction of Kinase Inhibitors: Toward the Virtual Assay","volume":"60","author":"Merget","year":"2017","journal-title":"J. Med. Chem."},{"key":"2026081804380887500_cit15","doi-asserted-by":"publisher","first-page":"156","DOI":"10.1021\/ci200314j","article-title":"Kinase-Kernel\nModels: Accurate in Silico Screening\nof 4 Million Compounds across the Entire Human Kinome","volume":"52","author":"Martin","year":"2012","journal-title":"J. Chem. Inf. Model."},{"key":"2026081804380887500_cit16","doi-asserted-by":"crossref","unstructured":"Zadrozny, B.; Elkan, C.\n          Transforming Classifier Scores\ninto Accurate Multiclass Probability Estimates. Proceedings of the Eighth ACM SIGKDD International Conference on\nKnowledge Discovery and Data Mining, 2002; pp 694\u2013699.","DOI":"10.1145\/775047.775151"},{"key":"2026081804380887500_cit17","doi-asserted-by":"crossref","unstructured":"Niculescu-Mizil, A.; Caruana, R.\n          Predicting\nGood Probabilities with Supervised Learning. Proceedings of the 22nd International Conference on Machine Learning, 2005; pp 625\u2013632.","DOI":"10.1145\/1102351.1102430"},{"key":"2026081804380887500_cit18","doi-asserted-by":"crossref","unstructured":"Ru\u0308ping, S.\n          \n          Robust\nProbabilistic Calibration. ECML, 2006; Springer, 2006; pp 743\u2013750.","DOI":"10.1007\/11871842_75"},{"key":"2026081804380887500_cit19","first-page":"1","article-title":"Classifier\nCalibration","volume-title":"Encyclopedia of Machine Learning\nand Data Mining","author":"Flach","year":"2016"},{"key":"2026081804380887500_cit20","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1007\/978-3-0348-8645-1_4","article-title":"Independent Identically\nDistributed (Iid) Random Variables","volume-title":"Discrete\nStochastics","author":"Jacobs","year":"1992"},{"key":"2026081804380887500_cit21","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1021\/acschembio.6b00706","article-title":"What Can We Learn from\nBioactivity Data? Chemoinformatics Tools and\nApplications in Chemical Biology Research","volume":"12","author":"Humbeck","year":"2017","journal-title":"ACS\nChem. Biol."},{"key":"2026081804380887500_cit22","doi-asserted-by":"publisher","first-page":"613","DOI":"10.3389\/fphar.2018.00613","article-title":"Extending\nin Silico Protein Target Prediction Models to Include Functional Effects","volume":"9","author":"Mervin","year":"2018","journal-title":"Front. Pharmacol."},{"key":"2026081804380887500_cit23","doi-asserted-by":"publisher","first-page":"2554","DOI":"10.1016\/j.jprot.2011.05.011","article-title":"From in Silico\nTarget Prediction\nto Multi-Target Drug Design: Current Databases, Methods and Applications","volume":"74","author":"Koutsoukas","year":"2011","journal-title":"J. Proteomics"},{"key":"2026081804380887500_cit24","doi-asserted-by":"publisher","first-page":"3007","DOI":"10.1021\/acschembio.6b00538","article-title":"Understanding\nCytotoxicity and Cytostaticity in a High-Throughput Screening Collection","volume":"11","author":"Mervin","year":"2016","journal-title":"ACS Chem. Biol."},{"key":"2026081804380887500_cit25","doi-asserted-by":"publisher","first-page":"1957","DOI":"10.1021\/ci300435j","article-title":"In Silico Target Predictions:\nDefining a Benchmarking Data Set and Comparison of Performance of\nthe Multiclass Nai\u0308ve Bayes and Parzen-Rosenblatt Window","volume":"53","author":"Koutsoukas","year":"2013","journal-title":"J. Chem. Inf. Model."},{"key":"2026081804380887500_cit26","doi-asserted-by":"publisher","first-page":"323","DOI":"10.2174\/1386207318666150305145012","article-title":"Comparing\nGlobal and Local Likelihood Score Thresholds\nin Multiclass Laplacian-Modified Naive Bayes Protein Target Prediction","volume":"18","author":"Drakakis","year":"2015","journal-title":"Comb. Chem. High Throughput Screen."},{"key":"2026081804380887500_cit27","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1186\/1758-2946-5-17","article-title":"The Influence\nof the Inactives Subset\nGeneration on the Performance of Machine Learning Methods","volume":"5","author":"Smusz","year":"2013","journal-title":"J. Cheminf."},{"key":"2026081804380887500_cit28","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1021\/ci025554v","article-title":"Dise: Directed Sphere\nExclusion","volume":"43","author":"Gobbi","year":"2003","journal-title":"J. Chem. Inf. Comput. Sci."},{"key":"2026081804380887500_cit29","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1186\/s13321-019-0388-x","article-title":"Large Scale Comparison of Qsar and Conformal Prediction\nMethods and Their Applications in Drug Discovery","volume":"11","author":"Bosc","year":"2019","journal-title":"J. Cheminf."},{"key":"2026081804380887500_cit30","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1186\/1758-2946-6-32","article-title":"The Influence\nof Negative Training\nSet Size on Machine Learning-Based Virtual Screening","volume":"6","author":"Kurczab","year":"2014","journal-title":"J. Cheminf."},{"key":"2026081804380887500_cit31","doi-asserted-by":"publisher","first-page":"710","DOI":"10.1021\/acs.jcim.7b00088","article-title":"Influence of Varying Training Set\nComposition and Size on Support Vector Machine-Based Prediction of\nActive Compounds","volume":"57","author":"Rodri\u0301guez-Pe\u0301rez","year":"2017","journal-title":"J. Chem. Inf. Model."},{"key":"2026081804380887500_cit32","doi-asserted-by":"publisher","first-page":"1591","DOI":"10.1021\/acs.jcim.7b00159","article-title":"Applying Mondrian Cross-Conformal Prediction to Estimate Prediction\nConfidence on Large Imbalanced Bioactivity Data Sets","volume":"57","author":"Sun","year":"2017","journal-title":"J. Chem. Inf. Model."},{"key":"2026081804380887500_cit33","doi-asserted-by":"publisher","first-page":"512","DOI":"10.1021\/ci050352v","article-title":"Assessing the Scaffold Diversity\nof Screening Libraries","volume":"46","author":"Krier","year":"2006","journal-title":"J. Chem. Inf. Model."},{"key":"2026081804380887500_cit34","doi-asserted-by":"publisher","first-page":"2174","DOI":"10.1021\/ci2001428","article-title":"Scaffold Diversity of Exemplified\nMedicinal Chemistry Space","volume":"51","author":"Langdon","year":"2011","journal-title":"J. Chem. Inf. Model."},{"key":"2026081804380887500_cit35","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1093\/bioinformatics\/btx525","article-title":"Orthologue\nChemical Space and Its Influence on Target Prediction","volume":"34","author":"Mervin","year":"2018","journal-title":"Bioinformatics"},{"key":"2026081804380887500_cit36","doi-asserted-by":"publisher","first-page":"189","DOI":"10.2174\/138620707780126705","article-title":"Target Specific Compound\nIdentification Using a Support Vector Machine","volume":"10","author":"Plewczynski","year":"2007","journal-title":"Comb. Chem. High Throughput Screen."},{"key":"2026081804380887500_cit37","doi-asserted-by":"crossref","unstructured":"Platt, J.\n          \n          Probabilistic\nOutputs for Support Vector Machines and Comparisons to Regularized\nLikelihood Methods. Advances in Large Margin\nClassifiers, 1999; Vol. 10,\npp 61\u201374.","DOI":"10.7551\/mitpress\/1113.003.0008"},{"key":"2026081804380887500_cit38","volume-title":"Naive Bayes Classifiers","author":"Murphy","year":"2006"},{"key":"2026081804380887500_cit39","unstructured":"Zhang, H.\n          \n          The Optimality\nof Naive Bayes. Proceedings of 17th International\nFlorida Artificial Intelligence Research Society Conference, 2004."},{"key":"2026081804380887500_cit40","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/a:1010933404324","article-title":"Random Forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"2026081804380887500_cit41","doi-asserted-by":"publisher","first-page":"3949","DOI":"10.1002\/sim.6959","article-title":"Calibrating Random\nForests for Probability Estimation","volume":"35","author":"Dankowski","year":"2016","journal-title":"Stat. Med."},{"key":"2026081804380887500_cit42","unstructured":"Guo, C.; Pleiss, G.; Sun, Y.; Weinberger, K. Q.\n          On Calibration of Modern Neural\nNetworks. Proceedings of the 34th International\nConference on Machine Learning, 2017; Vol. 70, pp 1321\u20131330."},{"key":"2026081804380887500_cit43","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/IJCNN.2019.8851962","article-title":"Are Traditional\nNeural Networks Well-Calibrated?","volume-title":"2019 International\nJoint Conference on Neural Networks (IJCNN)","author":"Johansson","year":"2019"},{"key":"2026081804380887500_cit44","doi-asserted-by":"publisher","first-page":"468","DOI":"10.1002\/wcms.1183","article-title":"Machine\nLearning Methods in Chemoinformatics","volume":"4","author":"Mitchell","year":"2014","journal-title":"Wiley\nInterdiscip. Rev. Comput. Mol. Sci."},{"key":"2026081804380887500_cit45","doi-asserted-by":"publisher","first-page":"323","DOI":"10.2174\/1386207318666150305145012","article-title":"Comparing Global and Local Likelihood Score Thresholds\nin Multiclass Laplacian-Modified Nai\u0308ve Bayes Protein Target\nPrediction","volume":"18","author":"Drakakis","year":"2015","journal-title":"Comb. Chem. High Throughput Screen."},{"key":"2026081804380887500_cit46","doi-asserted-by":"publisher","first-page":"230","DOI":"10.1186\/s12916-019-1466-7","article-title":"Calibration: The Achilles Heel of\nPredictive Analytics","volume":"17","author":"Van\nCalster","year":"2019","journal-title":"BMC Med."},{"key":"2026081804380887500_cit47","article-title":"Evaluating\nModel Calibration in Classification","author":"Vaicenavicius","year":"2019"},{"key":"2026081804380887500_cit48","doi-asserted-by":"publisher","first-page":"839","DOI":"10.1021\/ci0500381","article-title":"A Stepwise Approach\nfor Defining the Applicability Domain of Sar and Qsar Models","volume":"45","author":"Dimitrov","year":"2005","journal-title":"J. Chem. Inf. Model."},{"key":"2026081804380887500_cit49","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1186\/s13321-016-0182-y","article-title":"A Novel Applicability\nDomain Technique for Mapping Predictive Reliability across the Chemical\nSpace of a Qsar: Reliability-Density Neighbourhood","volume":"8","author":"Aniceto","year":"2016","journal-title":"J. Cheminf."},{"key":"2026081804380887500_cit50","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1016\/j.yrtph.2014.12.021","article-title":"Introducing Conformal\nPrediction in Predictive Modeling for Regulatory Purposes. A Transparent\nand Flexible Alternative to Applicability Domain Determination","volume":"71","author":"Norinder","year":"2015","journal-title":"Regul. Toxicol. Pharmacol."},{"key":"2026081804380887500_cit51","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1007\/s10472-013-9378-2","article-title":"The Application\nof\nConformal Prediction to the Drug Discovery Process","volume":"74","author":"Eklund","year":"2015","journal-title":"Ann. Math. Artif. Intell."},{"key":"2026081804380887500_cit52","doi-asserted-by":"publisher","first-page":"1596","DOI":"10.1021\/ci5001168","article-title":"Introducing\nConformal\nPrediction in Predictive Modeling. A Transparent and Flexible Alternative\nto Applicability Domain Determination","volume":"54","author":"Norinder","year":"2014","journal-title":"J. Chem.\nInf. Model."},{"key":"2026081804380887500_cit53","doi-asserted-by":"crossref","unstructured":"Vovk, V.\n          \n          Transductive Conformal\nPredictors. IFIP International Conference\non Artificial Intelligence Applications and Innovations, 2013; pp 348\u2013360.","DOI":"10.1007\/978-3-642-41142-7_36"},{"key":"2026081804380887500_cit54","doi-asserted-by":"crossref","DOI":"10.5772\/6078","article-title":"Inductive\nConformal Prediction: Theory and Application to Neural Networks","volume-title":"Tools in Artificial Intelligence","author":"Papadopoulos","year":"2008"},{"key":"2026081804380887500_cit55","unstructured":"Carlsson, L.; Eklund, M.; Norinder, U.\n          Aggregated Conformal\nPrediction. IFIP International Conference\non Artificial Intelligence Applications and Innovations, 2014; pp 231-240."},{"key":"2026081804380887500_cit56","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1007\/s10472-013-9368-4","article-title":"Cross-Conformal\nPredictors","volume":"74","author":"Vovk","year":"2015","journal-title":"Ann. Math. Artif. Intell."},{"key":"2026081804380887500_cit57","article-title":"Venn Predictors\nand Isotonic Regression","author":"Vovk","year":"2012"},{"key":"2026081804380887500_cit58","article-title":"Venn-Abers\nPredictors","author":"Vovk","year":"2012"},{"key":"2026081804380887500_cit59","unstructured":"Franc, V.; Zien, A.; Scho\u0308lkopf, B.\n          Support Vector Machines as\nProbabilistic Models. Proceedings of the 28th\nInternational Conference on Machine Learning (ICML-11), 2011; pp 665\u2013672."},{"key":"2026081804380887500_cit60","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1039\/c6tx00252h","article-title":"Modelling Compound\nCytotoxicity Using\nConformal Prediction and Pubchem Hts Data","volume":"6","author":"Svensson","year":"2017","journal-title":"Toxicol.\nRes."},{"key":"2026081804380887500_cit61","doi-asserted-by":"publisher","first-page":"80","DOI":"10.3389\/fenvs.2015.00080","article-title":"Deeptox: Toxicity\nPrediction Using Deep Learning","volume":"3","author":"Mayr","year":"2016","journal-title":"Front. Environ.\nSci."},{"key":"2026081804380887500_cit62","doi-asserted-by":"publisher","first-page":"641","DOI":"10.1214\/aoms\/1177728423","article-title":"An Empirical\nDistribution Function for Sampling with Incomplete Information","volume":"26","author":"Ayer","year":"1955","journal-title":"Ann. Math. Stat."},{"key":"2026081804380887500_cit63","doi-asserted-by":"publisher","first-page":"5052","DOI":"10.1214\/17-ejs1338si","article-title":"Beyond Sigmoids: How to Obtain Well-Calibrated\nProbabilities from Binary Classifiers with Beta Calibration","volume":"11","author":"Kull","year":"2017","journal-title":"Electron. J. Stat."},{"key":"2026081804380887500_cit64","doi-asserted-by":"crossref","unstructured":"Vezhnevets, A.; Barinova, O.\n          Avoiding\nBoosting Overfitting by Removing Confusing Samples. Machine Learning: ECML 2007, 2007; pp 430\u2013441.","DOI":"10.1007\/978-3-540-74958-5_40"},{"key":"2026081804380887500_cit65","unstructured":"Vovk, V.; Petej, I.; Fedorova, V.\n          Large-Scale Probabilistic Predictors\nwith and without Guarantees of Validity. Advances\nin Neural Information Processing Systems, 2015; pp 892\u2013900."},{"key":"2026081804380887500_cit66","unstructured":"Zhou, C.\n          \n          Conformal and\nVenn Predictors for Multi-Probabilistic Predictions and Their Applications. Ph.D. Thesis, Department of Computer Science, Royal\nHolloway, University of London, 2015."},{"key":"2026081804380887500_cit67","doi-asserted-by":"crossref","unstructured":"Vovk, V.; Petej, I.; Fedorova, V.\n          From\nConformal to Probabilistic\nPrediction. IFIP International Conference\non Artificial Intelligence Applications and Innovations, 2014; pp 221\u2013230.","DOI":"10.1007\/978-3-662-44722-2_24"},{"key":"2026081804380887500_cit68","unstructured":"Toccaceli, P.; Nouretdinov, I.; Luo, Z.; Vovk, V.; Carlsson, L.; Gammerman, A.\n          Excape Wp1. Probabilistic Prediction; Royal Holloway, 2016."},{"key":"2026081804380887500_cit69","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1007\/s10472-014-9420-z","article-title":"Inductive Venn Prediction","volume":"74","author":"Lambrou","year":"2015","journal-title":"Ann.\nMath. Artif. Intell."},{"key":"2026081804380887500_cit70","unstructured":"Manokhin, V.\n          \n          Multi-Class Probabilistic\nClassification Using Inductive and Cross Venn\u2013Abers Predictors. Conformal and Probabilistic Prediction and Applications, 2017; pp 228\u2013240."},{"key":"2026081804380887500_cit71","unstructured":"Vovk, V.; Petej, I.\n          Proceedings\nof the Thirtieth Conference\non Uncertainty in Artificial Intelligence; AUAI Press: Quebec City, Quebec,\nCanada, 2014; pp 829\u2013838."},{"key":"2026081804380887500_cit72","unstructured":"Arvidsson, S.; Spjuth, O.; Carlsson, L.; Toccaceli, P.\n          Prediction of\nMetabolic Transformations Using Cross Venn-Abers Predictors. Conformal and Probabilistic Prediction and Applications,\n2017, 2017; pp 118\u2013131."},{"key":"2026081804380887500_cit73","unstructured":"Thomas, J.\n          \n          Exascale Compound Activity\nPrediction Engine, 2014."},{"key":"2026081804380887500_cit74","doi-asserted-by":"publisher","first-page":"1230","DOI":"10.1021\/acs.jcim.8b00724","article-title":"Accurate Hit Estimation\nfor Iterative Screening Using Venn-Abers Predictors","volume":"59","author":"Buendia","year":"2019","journal-title":"J. Chem. Inf. Model."},{"key":"2026081804380887500_cit75","unstructured":"Ahlberg, E.; Buendia, R.; Carlsson, L.\n          Using Venn-Abers\nPredictors to Assess Cardio-Vascular Risk. Conformal and Probabilistic Prediction and Applications, 2018; pp 132\u2013146."},{"key":"2026081804380887500_cit76","doi-asserted-by":"publisher","first-page":"103350","DOI":"10.1016\/j.jbi.2019.103350","article-title":"Targeting the Uncertainty of Predictions\nat Patient-Level\nUsing an Ensemble of Classifiers Coupled with Calibration Methods,\nVenn-Abers, and Conformal Predictors: A Case Study in Ad","volume":"101","author":"Pereira","year":"2020","journal-title":"J. Biomed. Inf."},{"key":"2026081804380887500_cit77","doi-asserted-by":"publisher","first-page":"1119","DOI":"10.3390\/molecules22071119","article-title":"Prediction\nof Drug-Target Interaction Networks from the Integration of Protein\nSequences and Drug Chemical Structures","volume":"22","author":"Meng","year":"2017","journal-title":"Molecules"},{"key":"2026081804380887500_cit78","doi-asserted-by":"crossref","unstructured":"Fakhraei, S.; Raschid, L.; Getoor, L.\n          Proceedings\nof the 12th\nInternational Workshop on Data Mining in Bioinformatics; ACM: Chicago, Illinois, 2013; pp 10\u201317.","DOI":"10.1145\/2500863.2500870"},{"key":"2026081804380887500_cit79","doi-asserted-by":"publisher","first-page":"1019","DOI":"10.1016\/j.drudis.2011.10.005","article-title":"Making\nEvery Sar Point Count: The Development of Chemistry Connect for the\nLarge-Scale Integration of Structure and Bioactivity Data","volume":"16","author":"Muresan","year":"2011","journal-title":"Drug Discov. Today"},{"key":"2026081804380887500_cit80","doi-asserted-by":"publisher","first-page":"W623","DOI":"10.1093\/nar\/gkp456","article-title":"Pubchem: A Public Information System for Analyzing\nBioactivities of Small Molecules","volume":"37","author":"Wang","year":"2009","journal-title":"Nucleic Acids\nRes."},{"key":"2026081804380887500_cit81","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1016\/s1574-1400(08)00012-1","article-title":"Pubchem: Integrated\nPlatform of Small Molecules and Biological Activities","volume":"4","author":"Bolton","year":"2008","journal-title":"Annu. Rep. Comput. Chem."},{"key":"2026081804380887500_cit82","doi-asserted-by":"publisher","first-page":"D1083","DOI":"10.1093\/nar\/gkt1031","article-title":"The Chembl\nBioactivity Database:\nAn Update","volume":"42","author":"Bento","year":"2014","journal-title":"Nucleic Acids Res."},{"key":"2026081804380887500_cit83","doi-asserted-by":"publisher","first-page":"D8","DOI":"10.1093\/nar\/gks1189","article-title":"Database\nResources of the National Center for Biotechnology Information","volume":"41","author":"Coordinators","year":"2013","journal-title":"Nucleic Acids Res."},{"key":"2026081804380887500_cit84","unstructured":"OEChem, Version 2.0.0; OpenEye Scientific\nSoftware: Santa Fe, Nm."},{"key":"2026081804380887500_cit85","unstructured":"Landrum, G.\n          \n          Rdkit: Open-Source\nCheminformatics. 2012, http:\/\/www.rdkit.org accessed on\n28\/07\/2020."},{"key":"2026081804380887500_cit86","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1021\/c160017a018","article-title":"The Generation\nof a Unique Machine Description for Chemical Structures-a Technique\nDeveloped at Chemical Abstracts Service","volume":"5","author":"Morgan","year":"1965","journal-title":"J.\nChem. Doc."},{"key":"2026081804380887500_cit87","first-page":"2825","article-title":"Scikit-Learn:\nMachine Learning in\nPython","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"2026081804380887500_cit88","doi-asserted-by":"publisher","first-page":"283","DOI":"10.1175\/1520-0469(1950)007&lt;0283:tstota&gt;2.0.co;2","article-title":"The Statistical\nTheory of Turbulence and the Problem of Diffusion in the Atmosphere","volume":"7","author":"Brier","year":"1950","journal-title":"J. Meteorol."},{"key":"2026081804380887500_cit89","volume-title":"Forecast Verification:\nA Practitioner\u2019s Guide in Atmospheric Science","author":"Jolliffe","year":"2003"},{"key":"2026081804380887500_cit90","doi-asserted-by":"crossref","unstructured":"Cohen, I.; Goldszmidt, M.\n          Properties\nand\nBenefits of Calibrated Classifiers. European\nConference on Principles of Data Mining and Knowledge Discovery, 2004; pp 125\u2013136.","DOI":"10.1007\/978-3-540-30116-5_14"},{"key":"2026081804380887500_cit91","unstructured":"Nouretdinov, I.; Volkhonskiy, D.; Lim, P.; Toccaceli, P.; Gammerman, A.\n          Inductive\nVenn-Abers\nPredictive Distribution. Conformal and Probabilistic\nPrediction and Applications, 2018; pp 15\u201336."},{"key":"2026081804380887500_cit92","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1186\/s13321-017-0230-2","article-title":"Efficiency\nof Different Measures for Defining the Applicability Domain of Classification\nModels","volume":"9","author":"Klingspohn","year":"2017","journal-title":"J. Cheminf."},{"key":"2026081804380887500_cit93","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1109\/tssc.1970.300339","article-title":"The Nearest\nNeighbor Classification Rule with a Reject Option","volume":"6","author":"Hellman","year":"1970","journal-title":"IEEE Trans. Syst. Sci. Cybern."},{"key":"2026081804380887500_cit94","doi-asserted-by":"publisher","first-page":"2313","DOI":"10.1021\/ci800079x","article-title":"Ligand-Target\nPrediction Using Winnow and Naive Bayesian Algorithms and the Implications\nof Overall Performance Statistics","volume":"48","author":"Nigsch","year":"2008","journal-title":"J. Chem.\nInf. Model."},{"key":"2026081804380887500_cit95","doi-asserted-by":"publisher","first-page":"2190","DOI":"10.1021\/ci9000376","article-title":"Target Fishing for\nChemical Compounds Using Target-Ligand\nActivity Data and Ranking Based Methods","volume":"49","author":"Wale","year":"2009","journal-title":"J.\nChem. Inf. Model."},{"key":"2026081804380887500_cit96","doi-asserted-by":"publisher","first-page":"363","DOI":"10.1186\/1471-2105-9-363","article-title":"Virtual Screening\nof Gpcrs: An in Silico Chemogenomics Approach","volume":"9","author":"Jacob","year":"2008","journal-title":"BMC Bioinf."},{"key":"2026081804380887500_cit97","unstructured":"Nouretdinov, I.\n          \n          Improving\nReliable Probabilistic Prediction by Using Additional Knowledge. Conformal and Probabilistic Prediction and Applications, 2017; pp 193\u2013200."}],"container-title":["Journal of Chemical Information and Modeling"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pubs.acs.org\/doi\/pdf\/10.1021\/acs.jcim.0c00476","content-type":"application\/pdf","content-version":"vor","intended-application":"unspecified"},{"URL":"https:\/\/pubs.acs.org\/jcisd8\/article-pdf\/60\/10\/4546\/8897937\/ci0c00476.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/pubs.acs.org\/jcisd8\/article-pdf\/60\/10\/4546\/8897937\/ci0c00476.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T09:43:29Z","timestamp":1787046209000},"score":1,"resource":{"primary":{"URL":"https:\/\/pubs.acs.org\/jcisd8\/article\/60\/10\/4546\/849675\/Comparison-of-Scaling-Methods-to-Obtain-Calibrated"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,31]]},"references-count":97,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2020,8,31]]},"published-print":{"date-parts":[[2020,10,26]]}},"URL":"https:\/\/doi.org\/10.1021\/acs.jcim.0c00476","relation":{"has-preprint":[{"id-type":"doi","id":"10.26434\/chemrxiv.11526132.v2","asserted-by":"object"}]},"ISSN":["1549-9596","1549-960X"],"issn-type":[{"value":"1549-9596","type":"print"},{"value":"1549-960X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,31]]}}}