{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T15:28:21Z","timestamp":1785511701458,"version":"3.56.0"},"reference-count":37,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,2,20]],"date-time":"2024-02-20T00:00:00Z","timestamp":1708387200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,2,20]],"date-time":"2024-02-20T00:00:00Z","timestamp":1708387200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000865","name":"Bill and Melinda Gates Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000865","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"abstract":"Abstract<\/jats:title>Background<\/jats:title>The two-way partial AUC has been recently proposed as a way to directly quantify partial area under the ROC curve with simultaneous restrictions on the sensitivity and specificity ranges of diagnostic tests or classifiers. The metric, as originally implemented in the R package, is estimated using a nonparametric estimator based on a trimmed Mann-Whitney U-statistic, which becomes computationally expensive in large sample sizes. (Its computational complexity is of order$$O(n_x n_y)$$<\/jats:tex-math>O<\/mml:mi>(<\/mml:mo>n<\/mml:mi>x<\/mml:mi><\/mml:msub>n<\/mml:mi>y<\/mml:mi><\/mml:msub>)<\/mml:mo><\/mml:mrow><\/mml:math><\/jats:alternatives><\/jats:inline-formula>, where$$n_x$$<\/jats:tex-math>n<\/mml:mi>x<\/mml:mi><\/mml:msub><\/mml:math><\/jats:alternatives><\/jats:inline-formula>and$$n_y$$<\/jats:tex-math>n<\/mml:mi>y<\/mml:mi><\/mml:msub><\/mml:math><\/jats:alternatives><\/jats:inline-formula>represent the number of positive and negative cases, respectively). This is problematic since the statistical methodology for comparing estimates generated from alternative diagnostic tests\/classifiers relies on bootstrapping resampling and requires repeated computations of the estimator on a large number of bootstrap samples.<\/jats:p><\/jats:sec>Methods<\/jats:title>By leveraging the graphical and probabilistic representations of the AUC, partial AUCs, and two-way partial AUC, we derive a novel estimator for the two-way partial AUC, which can be directly computed from the output of any software able to compute AUC and partial AUCs. We implemented our estimator using the computationally efficient R package, which leverages a nonparametric approach using the trapezoidal rule for the computation of AUC and partial AUC scores. (Its computational complexity is of order$$O(n \\log n)$$<\/jats:tex-math>O<\/mml:mi>(<\/mml:mo>n<\/mml:mi>log<\/mml:mo>n<\/mml:mi>)<\/mml:mo><\/mml:mrow><\/mml:math><\/jats:alternatives><\/jats:inline-formula>, where$$n = n_x + n_y$$<\/jats:tex-math>n<\/mml:mi>=<\/mml:mo>n<\/mml:mi>x<\/mml:mi><\/mml:msub>+<\/mml:mo>n<\/mml:mi>y<\/mml:mi><\/mml:msub><\/mml:mrow><\/mml:math><\/jats:alternatives><\/jats:inline-formula>.). We compare the empirical bias and computation time of the proposed estimator against the original estimator provided in the package in a series of simulation studies and on two real datasets.<\/jats:p><\/jats:sec>Results<\/jats:title>Our estimator tended to be less biased than the original estimator based on the trimmed Mann-Whitney U-statistic across all experiments (and showed considerably less bias in the experiments based on small sample sizes). But, most importantly, because the computational complexity of the proposed estimator is of order$$O(n \\log n)$$<\/jats:tex-math>O<\/mml:mi>(<\/mml:mo>n<\/mml:mi>log<\/mml:mo>n<\/mml:mi>)<\/mml:mo><\/mml:mrow><\/mml:math><\/jats:alternatives><\/jats:inline-formula>, rather than$$O(n_x n_y)$$<\/jats:tex-math>O<\/mml:mi>(<\/mml:mo>n<\/mml:mi>x<\/mml:mi><\/mml:msub>n<\/mml:mi>y<\/mml:mi><\/mml:msub>)<\/mml:mo><\/mml:mrow><\/mml:math><\/jats:alternatives><\/jats:inline-formula>, it is much faster to compute when sample sizes are large.<\/jats:p><\/jats:sec>Conclusions<\/jats:title>The proposed estimator provides an improvement for the computation of two-way partial AUC, and allows the comparison of diagnostic tests\/machine learning classifiers in large datasets where repeated computations of the original estimator on bootstrap samples become too expensive to compute.<\/jats:p><\/jats:sec>","DOI":"10.1186\/s12911-023-02382-2","type":"journal-article","created":{"date-parts":[[2024,2,20]],"date-time":"2024-02-20T09:03:20Z","timestamp":1708419800000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["A novel estimator for the two-way partial AUC"],"prefix":"10.1186","volume":"24","author":[{"given":"Elias","family":"Chaibub Neto","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vijay","family":"Yadav","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Solveig K.","family":"Sieberts","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Larsson","family":"Omberg","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,2,20]]},"reference":[{"key":"2382_CR1","doi-asserted-by":"publisher","DOI":"10.1093\/oso\/9780198509844.001.0001","volume-title":"The statistical evaluation of medical tests for classification and prediction","author":"MS Pepe","year":"2003","unstructured":"Pepe MS. The statistical evaluation of medical tests for classification and prediction. USA: Oxford University Press; 2003."},{"key":"2382_CR2","doi-asserted-by":"publisher","first-page":"255","DOI":"10.1126\/science.aaa8415","volume":"349","author":"MI Jordan","year":"2015","unstructured":"Jordan MI, Mitchell TM. Machine learning: Trends, perspectives, and prospects. Science. 2015;349:255\u201360.","journal-title":"Science."},{"key":"2382_CR3","doi-asserted-by":"crossref","unstructured":"Doi K. Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comput Med Imaging Graph. 2007;31:198-201.","DOI":"10.1016\/j.compmedimag.2007.02.002"},{"key":"2382_CR4","doi-asserted-by":"publisher","first-page":"2136","DOI":"10.1016\/j.compbiomed.2013.10.007","volume":"43","author":"MRK Mookiah","year":"2013","unstructured":"Mookiah MRK, Acharya UR, Chua CK, Lim CM, Ng EYK, Laude A. Computer-aided diagnosis of diabetic retinopathy: a review. Comput Biol Med. 2013;43:2136\u201355.","journal-title":"Comput Biol Med."},{"key":"2382_CR5","first-page":"355","volume":"30","author":"VM Goncalves","year":"2014","unstructured":"Goncalves VM, Delamaro ME, Nunes FLS. A systematic review on the evaluation and characteristics of computer-aided diagnosis systems. Braz J Biomed Eng. 2014;30:355\u201383.","journal-title":"Braz J Biomed Eng."},{"key":"2382_CR6","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1007\/s12194-019-00552-4","volume":"13","author":"H Fujita","year":"2020","unstructured":"Fujita H. AI-based computer-aided diagnosis (AI-CAD): the latest review to read first. Radiol Phys Technol. 2020;13:6\u201319.","journal-title":"Radiol Phys Technol."},{"issue":"5","key":"2382_CR7","doi-asserted-by":"publisher","first-page":"e218","DOI":"10.1002\/mp.13764","volume":"47","author":"HP Chan","year":"2020","unstructured":"Chan HP, Hadjiiski LM, Samala RK. Computer-aided diagnosis in the era of deep learning. Med Phys. 2020;47(5):e218\u201327. https:\/\/doi.org\/10.1002\/mp.13764.","journal-title":"Med Phys."},{"key":"2382_CR8","doi-asserted-by":"publisher","first-page":"861","DOI":"10.1016\/j.patrec.2005.10.010","volume":"27","author":"T Fawcett","year":"2006","unstructured":"Fawcett T. An introduction to ROC analysis. Pattern Recogn Lett. 2006;27:861\u201374. https:\/\/doi.org\/10.1016\/j.patrec.2005.10.010.","journal-title":"Pattern Recogn Lett."},{"key":"2382_CR9","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1002\/sim.4780060207","volume":"6","author":"K Linnet","year":"1987","unstructured":"Linnet K. Comparison of quantitative diagnostic tests type I error, power, and sample size. Stat Med. 1987;6:147\u201358.","journal-title":"Stat Med."},{"key":"2382_CR10","doi-asserted-by":"publisher","first-page":"1285","DOI":"10.1126\/science.3287615","volume":"240","author":"JA Swets","year":"1988","unstructured":"Swets JA. Measuring the accuracy of diagnostic systems. Science. 1988;240:1285\u201393.","journal-title":"Science"},{"key":"2382_CR11","first-page":"307","volume":"29","author":"JA Hanley","year":"1989","unstructured":"Hanley JA. Receiver operating characteristic (ROC) methodology: the state of the art. Crit Rev Diagn Imaging. 1989;29:307\u201335.","journal-title":"Crit Rev Diagn Imaging."},{"key":"2382_CR12","doi-asserted-by":"publisher","first-page":"1887","DOI":"10.1002\/sim.4780101205","volume":"10","author":"CB Begg","year":"1991","unstructured":"Begg CB. Advances in statistical methodology for diagnostic medicine in the 1980\u2019s. Stat Med. 1991;10:1887\u201395.","journal-title":"Stat Med."},{"key":"2382_CR13","doi-asserted-by":"publisher","DOI":"10.1002\/9780470906514","volume-title":"Statistical methods in diagnostic medicine","author":"XH Zhou","year":"2011","unstructured":"Zhou XH, Obuchowski NA, McClish DK. Statistical methods in diagnostic medicine. 2nd ed. Hoboken: John Wiley & Sons; 2011.","edition":"2"},{"key":"2382_CR14","unstructured":"World Health Organization. High priority target product profiles for new tuberculosis diagnostics: report of a consensus meeting, 28-29 April 2014, Geneva, Switzerland. World Health Organization; 2014. https:\/\/apps.who.int\/iris\/handle\/10665\/135617."},{"key":"2382_CR15","unstructured":"World Health Organization. Antigen-detection in the diagnosis of SARS-CoV-2 infection: Interim guidance, 6 October 2021. World Health Organization; 2021. https:\/\/iris.who.int\/handle\/10665\/345948."},{"issue":"1","key":"2382_CR16","doi-asserted-by":"publisher","first-page":"184","DOI":"10.1177\/0962280217718866","volume":"28","author":"H Yang","year":"2019","unstructured":"Yang H, Lu K, Lyu X, Hu F. Two-way partial AUC and its properties. Stat Methods Med Res. 2019;28(1):184\u201395.","journal-title":"Stat Methods Med Res."},{"key":"2382_CR17","doi-asserted-by":"publisher","first-page":"614","DOI":"10.1111\/1541-0420.00071","volume":"59","author":"LE Dodd","year":"2003","unstructured":"Dodd LE, Pepe MS. Partial AUC estimation and regression. Biometrics. 2003;59:614\u201323.","journal-title":"Biometrics."},{"key":"2382_CR18","doi-asserted-by":"publisher","first-page":"190","DOI":"10.1177\/0272989X8900900307","volume":"9","author":"DK McClish","year":"1989","unstructured":"McClish DK. Analysing a portion of the ROC curve. Med Dec Making. 1989;9:190\u20135.","journal-title":"Med Dec Making."},{"key":"2382_CR19","doi-asserted-by":"publisher","first-page":"701","DOI":"10.1002\/sim.1011","volume":"21","author":"DD Zhang","year":"2002","unstructured":"Zhang DD, Zhou XH, Freeman DH, et al. A non-parametric method for the comparison of partial areas under ROC curves and its application to large health care data sets. Stat Med. 2002;21:701\u201315.","journal-title":"Stat Med."},{"key":"2382_CR20","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1080\/19466315.2012.692514","volume":"4","author":"X Wang","year":"2012","unstructured":"Wang X, Ma J, George S, et al. Estimation of AUC or partial AUC under test-result-dependent sampling. Stat Biopharm Res. 2012;4:313\u201323.","journal-title":"Stat Biopharm Res."},{"key":"2382_CR21","doi-asserted-by":"publisher","first-page":"3449","DOI":"10.1002\/sim.5777","volume":"32","author":"H Ma","year":"2013","unstructured":"Ma H, Bandos AI, Rockette HE, et al. On use of partial area under the ROC curve for evaluation of diagnostic performance. Stat Med. 2013;32:3449\u201358.","journal-title":"Stat Med"},{"key":"2382_CR22","doi-asserted-by":"crossref","unstructured":"Mann HB, Whitney DR. On a test of whether one of two random variables is stochastically larger than the other. Ann Math Stat. 1947;18:50\u201360.","DOI":"10.1214\/aoms\/1177730491"},{"key":"2382_CR23","unstructured":"R Core Team. R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2021. https:\/\/www.R-project.org\/."},{"key":"2382_CR24","unstructured":"Efron B, Tibshirani R. An Introduction to the Bootstrap. Chapman & Hall\/CRC Monographs on Statistics and Applied Probability. Philadelphia: Chapman & Hall\/CRC; 1994."},{"key":"2382_CR25","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1186\/147121051277","volume":"12","author":"X Robin","year":"2011","unstructured":"Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 2011;12:77. https:\/\/doi.org\/10.1186\/147121051277.","journal-title":"BMC Bioinformatics."},{"key":"2382_CR26","doi-asserted-by":"publisher","first-page":"387","DOI":"10.1016\/0022-2496(75)90001-2","volume":"12","author":"D Bamber","year":"1975","unstructured":"Bamber D. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psychol. 1975;12:387\u2013415.","journal-title":"J Math Psychol."},{"key":"2382_CR27","doi-asserted-by":"publisher","first-page":"1277","DOI":"10.1002\/sim.4780081011","volume":"8","author":"ML Thompson","year":"1989","unstructured":"Thompson ML, Zucchini W. On the statistical analysis of ROC curves. Stat Med. 1989;8:1277\u201390.","journal-title":"Stat Med."},{"key":"2382_CR28","doi-asserted-by":"publisher","first-page":"745","DOI":"10.1148\/radiology.201.3.8939225","volume":"201","author":"Y Jiang","year":"1996","unstructured":"Jiang Y, Metz C, Nishikawa R. A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology. 1996;201:745\u201350.","journal-title":"Radiology."},{"key":"2382_CR29","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1016\/j.knosys.2013.12.023","volume":"60","author":"B Antal","year":"2014","unstructured":"Antal B, Hajdu A. An ensemble-based system for automatic screening of diabetic retinopathy. Knowl-Based Syst. 2014;60:20\u20137.","journal-title":"Knowl-Based Syst."},{"key":"2382_CR30","doi-asserted-by":"publisher","first-page":"17156","DOI":"10.1038\/s41598-020-73558-3","volume":"10","author":"D Chicco","year":"2020","unstructured":"Chicco D, Jurman G. Survival prediction of patients with sepsis from age, sex, and septic episode number alone. Sci Rep. 2020;10:17156.","journal-title":"Sci Rep."},{"key":"2382_CR31","unstructured":"Dua D, Graff C. UCI Machine Learning Repository [http:\/\/archive.ics.uci.edu\/ml]. Irvine: University of California, School of Information and Computer Science; 2019."},{"key":"2382_CR32","doi-asserted-by":"publisher","first-page":"231","DOI":"10.5566\/ias.1155","volume":"33","author":"E Decenciere","year":"2014","unstructured":"Decenciere E, Zhang X, Cazuguel G, Lay B, Cochener B, Trone C. Feedback on a publicly distributed image database: the Messidor database. Image Anal Stereol. 2014;33:231\u20134.","journal-title":"Image Anal Stereol."},{"issue":"2","key":"2382_CR33","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1111\/j.2517-6161.1958.tb00292.x","volume":"20","author":"DR Cox","year":"1958","unstructured":"Cox DR. The regression analysis of binary sequences. J R Stat Soc Ser B Methodol. 1958;20(2):215\u201332.","journal-title":"J R Stat Soc Ser B Methodol."},{"key":"2382_CR34","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/A:1010933404324","volume":"45","author":"L Breiman","year":"2001","unstructured":"Breiman L. Random forests. Mach Learn. 2001;45:5\u201332.","journal-title":"Mach Learn."},{"key":"2382_CR35","doi-asserted-by":"publisher","unstructured":"Wright MN, Ziegler A (2017). ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R. J Stat Softw. 2017;77:1-17. https:\/\/doi.org\/10.18637\/jss.v077.i01.","DOI":"10.18637\/jss.v077.i01"},{"key":"2382_CR36","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1148\/radiology.143.1.7063747","volume":"143","author":"JA Hanley","year":"1982","unstructured":"Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143:29\u201336.","journal-title":"Radiology."},{"issue":"1","key":"2382_CR37","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1177\/0272989X9701700111","volume":"17","author":"KO Hajian-Tilak","year":"1997","unstructured":"Hajian-Tilak KO, Hanley JA, Joseph L, Collet JP. A comparison of parametric and non-parametric approaches to ROC analysis of quantitative diagnostic tests. Med Decis Making. 1997;17(1):94\u2013102. https:\/\/doi.org\/10.1177\/0272989X9701700111.","journal-title":"Med Decis Making."}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-023-02382-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-023-02382-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-023-02382-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,11]],"date-time":"2024-11-11T22:47:03Z","timestamp":1731365223000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-023-02382-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,20]]},"references-count":37,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["2382"],"URL":"https:\/\/doi.org\/10.1186\/s12911-023-02382-2","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,20]]},"assertion":[{"value":"2 December 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 November 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 February 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"57"}}