{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,27]],"date-time":"2026-01-27T13:16:19Z","timestamp":1769519779085,"version":"3.49.0"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,7,28]],"date-time":"2022-07-28T00:00:00Z","timestamp":1658966400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,7,28]],"date-time":"2022-07-28T00:00:00Z","timestamp":1658966400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"crossref","award":["T32ES007329"],"award-info":[{"award-number":["T32ES007329"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Triangle Center of Evolutionary Medicine"},{"DOI":"10.13039\/100004784","name":"SAS Institute","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100004784","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Cheminform"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In virtual screening for drug discovery, hit enrichment curves are widely used to assess the performance of ranking algorithms with regard to their ability to identify early enrichment. Unfortunately, researchers almost never consider the uncertainty associated with estimating such curves before declaring differences between performance of competing algorithms. Uncertainty is often large because the testing fractions of interest to researchers are small. Appropriate inference is complicated by two sources of correlation that are often overlooked: correlation across different testing fractions within a single algorithm, and correlation between competing algorithms. Additionally, researchers are often interested in making comparisons along the entire curve, not only at a few testing fractions. We develop inferential procedures to address both the needs of those interested in a few testing fractions, as well as those interested in the entire curve. For the former, four hypothesis testing and (pointwise) confidence intervals are investigated, and a newly developed EmProc approach is found to be most effective. For inference along entire curves, EmProc-based confidence bands are recommended for simultaneous coverage and minimal width. While we focus on the hit enrichment curve, this work is also appropriate for lift curves that are used throughout the machine learning community. Our inferential procedures trivially extend to enrichment factors, as well.<\/jats:p>","DOI":"10.1186\/s13321-022-00629-0","type":"journal-article","created":{"date-parts":[[2022,7,28]],"date-time":"2022-07-28T18:15:19Z","timestamp":1659032119000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Confidence bands and hypothesis tests for hit enrichment curves"],"prefix":"10.1186","volume":"14","author":[{"given":"Jeremy R","family":"Ash","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jacqueline M","family":"Hughes-Oliver","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,7,28]]},"reference":[{"issue":"5","key":"629_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v028.i05","volume":"28","author":"M Kuhn","year":"2008","unstructured":"Kuhn M (2008) Building predictive models in R using the caret package. J Stat Softw 28(5):1\u201326. https:\/\/doi.org\/10.18637\/jss.v028.i05","journal-title":"J Stat Softw"},{"key":"629_CR2","unstructured":"SAS Institute Inc (2020) SAS Enterprise Miner 15.1, Cary, NC"},{"key":"629_CR3","unstructured":"SAS Institute Inc (2020) JMP 16.0, Cary, NC x"},{"issue":"2","key":"629_CR4","doi-asserted-by":"publisher","first-page":"205","DOI":"10.1021\/ci900419k","volume":"50","author":"H Geppert","year":"2010","unstructured":"Geppert H, Vogt M, Bajorath J (2010) Current trends in ligand-based virtual screening: Molecular representations, data mining methods, new application areas, and performance evaluation. J Chem Inf Model 50(2):205\u2013216. https:\/\/doi.org\/10.1021\/ci900419k","journal-title":"J Chem Inf Model"},{"key":"629_CR5","doi-asserted-by":"publisher","unstructured":"Rosset S, Neumann E, Eick U, Vatnik N, Idan I (2001) Evaluation of prediction models for marketing campaigns, pp 456\u2013461. ACM Press, New York, NY. https:\/\/doi.org\/10.1145\/502512.502581","DOI":"10.1145\/502512.502581"},{"key":"629_CR6","doi-asserted-by":"publisher","unstructured":"Empereur-Mot C, Zagury J-F, Montes M (2016) Screening explorer-an interactive tool for the analysis of screening results. J Chem Inf Model 56(12):2281\u20132286. https:\/\/doi.org\/10.1021\/acs.jcim.6b00283 ((Web application at http:\/\/stats.drugdesign.fr))","DOI":"10.1021\/acs.jcim.6b00283"},{"key":"629_CR7","unstructured":"NCBI (2021) https:\/\/www.ncbi.nlm.nih.gov\/gene\/5468"},{"issue":"17","key":"629_CR8","doi-asserted-by":"publisher","first-page":"6560","DOI":"10.1021\/jm301916b","volume":"56","author":"T Zhu","year":"2013","unstructured":"Zhu T, Cao S, Su P-C, Patel R, Shah D, Chokshi HB, Szukala R, Johnson ME, Hevener KE (2013) Hit identification and optimization in virtual screening: Practical recommendations based on a critical literature analysis. J Med Chem 56(17):6560\u20136572. https:\/\/doi.org\/10.1021\/jm301916b","journal-title":"J Med Chem"},{"issue":"2","key":"629_CR9","doi-asserted-by":"publisher","first-page":"488","DOI":"10.1021\/ci600426e","volume":"47","author":"J-F Truchon","year":"2007","unstructured":"Truchon J-F, Bayly CI (2007) Evaluating virtual screening methods: good and bad metrics for the \u201cearly recognition\u201dproblem. J Chem Inf Model 47(2):488\u2013508. https:\/\/doi.org\/10.1021\/ci600426e","journal-title":"J Chem Inf Model"},{"issue":"3\u20134","key":"629_CR10","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1007\/s10822-008-9196-5","volume":"22","author":"AN Jain","year":"2008","unstructured":"Jain AN, Nicholls A (2008) Recommendations for evaluation of computational methods. J Comput Aided Mol Des 22(3\u20134):133\u2013139. https:\/\/doi.org\/10.1007\/s10822-008-9196-5","journal-title":"J Comput Aided Mol Des"},{"issue":"9","key":"629_CR11","doi-asserted-by":"publisher","first-page":"887","DOI":"10.1007\/s10822-014-9753-z","volume":"28","author":"A Nicholls","year":"2014","unstructured":"Nicholls A (2014) Confidence limits, error bars and method comparison in molecular modeling. Part 1: the calculation of confidence intervals. J Comput Aided Mol Des 28(9):887\u2013918","journal-title":"J Comput Aided Mol Des"},{"key":"629_CR12","doi-asserted-by":"publisher","DOI":"10.1007\/s10822-019-00274-0","author":"MC Robinson","year":"2020","unstructured":"Robinson MC, Glen RC, Lee AA (2020) Validating the validation: reanalyzing a large-scale comparison of deep learning and machine learning models for bioactivity prediction. J Comput Aided Mol Des. https:\/\/doi.org\/10.1007\/s10822-019-00274-0","journal-title":"J Comput Aided Mol Des"},{"issue":"3\u20134","key":"629_CR13","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1007\/s10822-007-9166-3","volume":"22","author":"PCD Hawkins","year":"2008","unstructured":"Hawkins PCD, Warren GL, Skillman AG, Nicholls A (2008) How to do an evaluation: pitfalls and traps. J Comput Aided Mol Des 22(3\u20134):179\u2013190. https:\/\/doi.org\/10.1007\/s10822-007-9166-3","journal-title":"J Comput Aided Mol Des"},{"issue":"512","key":"629_CR14","doi-asserted-by":"publisher","first-page":"1717","DOI":"10.1080\/01621459.2014.993080","volume":"110","author":"W Jiang","year":"2015","unstructured":"Jiang W, Zhao Y (2015) On asymptotic distributions and confidence intervals for lift measures in data mining. J Am Stat Assoc 110(512):1717\u20131725. https:\/\/doi.org\/10.1080\/01621459.2014.993080","journal-title":"J Am Stat Assoc"},{"issue":"7","key":"629_CR15","doi-asserted-by":"publisher","first-page":"1739","DOI":"10.1021\/jm0306430","volume":"47","author":"RA Friesner","year":"2004","unstructured":"Friesner RA, Banks JL, Murphy RB, Halgren TA, Klicic JJ, Mainz DT, Repasky MP, Knoll EH, Shelley M, Perry JK, Shaw DE, Francis P, Shenkin PS (2004) Glide: a new approach for rapid, accurate docking and scoring. 1. Method and assessment of docking accuracy. J Med Chem 47(7):1739\u20131749. https:\/\/doi.org\/10.1021\/jm0306430","journal-title":"J Med Chem"},{"key":"629_CR16","volume-title":"Nonparametric econometrics: theory and practice","author":"Q Li","year":"2007","unstructured":"Li Q, Racine JS (2007) Nonparametric econometrics: theory and practice. Princeton University Press, New York, NY"},{"key":"629_CR17","volume-title":"An introduction to categorical data analysis. Wiley series in probability and statistics","author":"A Agresti","year":"2007","unstructured":"Agresti A (2007) An introduction to categorical data analysis. Wiley series in probability and statistics. Wiley, Hoboken, NJ"},{"issue":"1","key":"629_CR18","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1186\/1471-2288-13-91","volume":"13","author":"MW Fagerland","year":"2013","unstructured":"Fagerland MW, Lydersen S, Laake P (2013) The McNemar test for binary matched-pairs data: mid-p and asymptotic are better than exact conditional. BMC Med Res Methodol 13(1):91. https:\/\/doi.org\/10.1186\/1471-2288-13-91","journal-title":"BMC Med Res Methodol"},{"key":"629_CR19","doi-asserted-by":"publisher","first-page":"2635","DOI":"10.1002\/(SICI)1097-0258(19981130)17:22<2635::AID-SIM954>3.0.CO;2-C","volume":"17","author":"RG Newcombe","year":"1998","unstructured":"Newcombe RG (1998) Improved confidence intervals for the difference between binomial proportions based on paired data. Stat Med 17:2635\u20132650. https:\/\/doi.org\/10.1002\/(SICI)1097-0258(19981130)17:22<2635::AID-SIM954>3.0.CO;2-C","journal-title":"Stat Med"},{"key":"629_CR20","unstructured":"Rodriguez\u00a0de Gil P, Pham JRT, Nguyen D, Kromrey JD, Kim ES ( 2013) SAS macros CORR-P and TANGO: interval estimation for the difference between correlated proportions in dependent samples. In: Proceedings of the SouthEast SAS Users Group 2013"},{"issue":"4","key":"629_CR21","doi-asserted-by":"publisher","first-page":"479","DOI":"10.3102\/1076998611411915","volume":"37","author":"DG Bonett","year":"2012","unstructured":"Bonett DG, Price RM (2012) Adjusted Wald confidence interval for a difference of binomial proportions based on paired data. J Educ Behav Stat 37(4):479\u2013488. https:\/\/doi.org\/10.3102\/1076998611411915","journal-title":"J Educ Behav Stat"},{"key":"629_CR22","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1016\/j.ymeth.2014.11.015","volume":"71","author":"J Xia","year":"2015","unstructured":"Xia J, Tilahun EL, Reid T-E, Zhang L, Wang XS (2015) Benchmarking methods and data sets for ligand enrichment assessment in virtual screening. Methods 71:146\u2013157. https:\/\/doi.org\/10.1016\/j.ymeth.2014.11.015","journal-title":"Methods"},{"issue":"34","key":"629_CR23","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1007\/s10822-007-9167-2","volume":"22","author":"AC Good","year":"2008","unstructured":"Good AC, Oprea TI (2008) Optimization of CAMD techniques 3. Virtual screening enrichment studies: a help or hindrance in tool selection? J Comput Aided Mol Des 22(34):169\u2013178. https:\/\/doi.org\/10.1007\/s10822-007-9167-2","journal-title":"J Comput Aided Mol Des"},{"key":"629_CR24","doi-asserted-by":"publisher","unstructured":"Stumpfe D, Bajorath J (2011) Applied virtual screening: strategies, recommendations, and caveats, pp 291\u2013 318 . https:\/\/doi.org\/10.1002\/9783527633326.ch11","DOI":"10.1002\/9783527633326.ch11"},{"issue":"6","key":"629_CR25","doi-asserted-by":"publisher","first-page":"1447","DOI":"10.1021\/ci400115b","volume":"53","author":"MR Bauer","year":"2013","unstructured":"Bauer MR, Ibrahim TM, Vogel SM, Boeckler FM (2013) Evaluation and optimization of virtual screening workflows with dekois 2.0\u2014a public library of challenging docking benchmark sets. J Chem Inf Model 53(6):1447\u20131462. https:\/\/doi.org\/10.1021\/ci400115b","journal-title":"J Chem Inf Model"},{"issue":"D1","key":"629_CR26","doi-asserted-by":"publisher","first-page":"945","DOI":"10.1093\/nar\/gkw1074","volume":"45","author":"A Gaulton","year":"2017","unstructured":"Gaulton A, Hersey A, Nowotka M, Bento AP, Chambers J, Mendez D, Mutowo P, Atkinson F, Bellis LJ, Cibri\u00e1n-Uhalte E, Davies M, Dedman N, Karlsson A, Magari\u00f1os MP, Overington J.P, Papadatos G, Smit I, Leach A.R (2017) The ChEMBL database in 2017. Nucleic Acids Res 45(D1):945\u2013954. https:\/\/doi.org\/10.1093\/nar\/gkw1074","journal-title":"Nucleic Acids Res"},{"issue":"11","key":"629_CR27","doi-asserted-by":"publisher","first-page":"2324","DOI":"10.1021\/acs.jcim.5b00559","volume":"55","author":"T Sterling","year":"2015","unstructured":"Sterling T, Irwin JJ (2015) Zinc 15\u2014ligand discovery for everyone. J Chem Inf Model 55(11):2324\u20132337. https:\/\/doi.org\/10.1021\/acs.jcim.5b00559","journal-title":"J Chem Inf Model"},{"issue":"3\u20134","key":"629_CR28","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1007\/s10822-008-9189-4","volume":"22","author":"JJ Irwin","year":"2008","unstructured":"Irwin JJ (2008) Community benchmarks for virtual screening. J Comput Aided Mol Des 22(3\u20134):193\u2013199. https:\/\/doi.org\/10.1007\/s10822-008-9189-4","journal-title":"J Comput Aided Mol Des"},{"issue":"14","key":"629_CR29","doi-asserted-by":"publisher","first-page":"6582","DOI":"10.1021\/jm300687e","volume":"55","author":"MM Mysinger","year":"2012","unstructured":"Mysinger MM, Carchia M, Irwin JJ, Shoichet BK (2012) Directory of useful decoys, enhanced (dud-e): better ligands and decoys for better benchmarking. J Med Chem 55(14):6582\u20136594. https:\/\/doi.org\/10.1021\/jm300687e","journal-title":"J Med Chem"},{"issue":"2","key":"629_CR30","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1021\/ci8002649","volume":"49","author":"SG Rohrer","year":"2009","unstructured":"Rohrer SG, Baumann K (2009) Maximum unbiased validation (muv) data sets for virtual screening based on pubchem bioactivity data. J Chem Inf Model 49(2):169\u2013184. https:\/\/doi.org\/10.1021\/ci8002649","journal-title":"J Chem Inf Model"},{"key":"629_CR31","doi-asserted-by":"publisher","unstructured":"NCBI Resource Coordinators ( 2016) Database resources of the national center for biotechnology information. Nucleic Acids Res 44(1), 7\u201319. https:\/\/doi.org\/10.1093\/nar\/gkv1290","DOI":"10.1093\/nar\/gkv1290"},{"key":"629_CR32","unstructured":"Hofert M, Kojadinovic I, Maechler M, Yan J (2020) Copula: multivariate dependence with copulas. R package version 1.0-1. https:\/\/CRAN.R-project.org\/package=copula"},{"key":"629_CR33","unstructured":"Jiang W, Zhao Y (2014) Some technical details on confidence intervals for lift measures in data mining. Technical report"},{"key":"629_CR34","volume-title":"Methods development for quantitative structure-activity relationships","author":"JR Ash","year":"2020","unstructured":"Ash JR (2020) Methods development for quantitative structure-activity relationships. North Carolina State University, Raleigh, NC"},{"issue":"1","key":"629_CR35","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1002\/jae.2656","volume":"34","author":"JL Montiel Olea","year":"2019","unstructured":"Montiel Olea JL, Plagborg-M\u00f8ller M (2019) Simultaneous confidence bands: Theory, implementation, and an application to svars. J Appl Economet 34(1):1\u201317. https:\/\/doi.org\/10.1002\/jae.2656","journal-title":"J Appl Economet"},{"issue":"2","key":"629_CR36","doi-asserted-by":"publisher","first-page":"119","DOI":"10.2307\/2685469","volume":"52","author":"A Agresti","year":"1998","unstructured":"Agresti A, Coull BA (1998) Approximate is better than \u2018exact\u2019 for interval estimation of binomial proportions. Am Stat 52(2):119\u2013126. https:\/\/doi.org\/10.2307\/2685469","journal-title":"Am Stat"},{"key":"629_CR37","doi-asserted-by":"publisher","first-page":"289","DOI":"10.2307\/2346101","volume":"57","author":"Y Benjamini","year":"1995","unstructured":"Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Ser B (Methodological) 57:289\u2013300. https:\/\/doi.org\/10.2307\/2346101","journal-title":"J R Stat Soc Ser B (Methodological)"},{"issue":"1","key":"629_CR38","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1214\/aoms\/1177729694","volume":"22","author":"S Kullback","year":"1951","unstructured":"Kullback S, Leibler RA (1951) On information and sufficiency. Ann Math Stat 22(1):79\u201386. https:\/\/doi.org\/10.1214\/aoms\/1177729694","journal-title":"Ann Math Stat"},{"issue":"3","key":"629_CR39","doi-asserted-by":"publisher","first-page":"0118432","DOI":"10.1371\/journal.pone.0118432","volume":"10","author":"T Saito","year":"2015","unstructured":"Saito T, Rehmsmeier M (2015) The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE 10(3):0118432. https:\/\/doi.org\/10.1371\/journal.pone.0118432","journal-title":"PLoS ONE"},{"issue":"5","key":"629_CR40","doi-asserted-by":"publisher","first-page":"1395","DOI":"10.1021\/ci0100144","volume":"41","author":"RP Sheridan","year":"2001","unstructured":"Sheridan RP, Singh SB, Fluder EM, Kearsley SK (2001) Protocols for bridging the peptide to nonpeptide gap in topological similarity searches. J Chem Inf Comput Sci 41(5):1395\u20131406. https:\/\/doi.org\/10.1021\/ci0100144","journal-title":"J Chem Inf Comput Sci"}],"container-title":["Journal of Cheminformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13321-022-00629-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13321-022-00629-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13321-022-00629-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,7,28]],"date-time":"2022-07-28T18:15:31Z","timestamp":1659032131000},"score":1,"resource":{"primary":{"URL":"https:\/\/jcheminf.biomedcentral.com\/articles\/10.1186\/s13321-022-00629-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,28]]},"references-count":40,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["629"],"URL":"https:\/\/doi.org\/10.1186\/s13321-022-00629-0","relation":{},"ISSN":["1758-2946"],"issn-type":[{"value":"1758-2946","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,28]]},"assertion":[{"value":"15 February 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 June 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 July 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"50"}}