{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,16]],"date-time":"2026-10-16T16:15:08Z","timestamp":1792167308723,"version":"4.4.0"},"reference-count":36,"publisher":"Wiley","issue":"23","license":[{"start":{"date-parts":[[2012,3,13]],"date-time":"2012-03-13T00:00:00Z","timestamp":1331596800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"},{"start":{"date-parts":[[2012,3,13]],"date-time":"2012-03-13T00:00:00Z","timestamp":1331596800000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"},{"start":{"date-parts":[[2012,3,13]],"date-time":"2012-03-13T00:00:00Z","timestamp":1331596800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"content-domain":{"domain":["wiley.com"],"crossmark-restriction":true},"short-container-title":["Statistics in Medicine"],"published-print":{"date-parts":[[2012,10,15]]},"abstract":"<jats:p>\n                    The area under the receiver operating characteristics curve (AUC of ROC) is a widely used measure of discrimination in risk prediction models. Routinely, the Mann\u2013Whitney statistics is used as an estimator of AUC, while the change in AUC is tested by the DeLong test. However, very often, in settings where the model is developed and tested on the same dataset, the added predictor is statistically significantly associated with the outcome but fails to produce a significant improvement in the AUC. No conclusive resolution exists to explain this finding. In this paper, we will show that the reason lies in the inappropriate application of the DeLong test in the setting of nested models. Using numerical simulations and a theoretical argument based on generalized\n                    <jats:italic>U<\/jats:italic>\n                    \u2010statistics, we show that if the added predictor is not statistically significantly associated with the outcome, the null distribution is non\u2010normal, contrary to the assumption of DeLong test. Our simulations of different scenarios show that the loss of power because of such a misuse of the DeLong test leads to a conservative test for small and moderate effect sizes. This problem does not exist in cases of predictors that are associated with the outcome and for non\u2010nested models. We suggest that for nested models, only the test of association be performed for the new predictors, and if the result is significant, change in AUC be estimated with an appropriate confidence interval, which can be based on the DeLong approach. Copyright \u00a9 2012 John Wiley &amp; Sons, Ltd.\n                  <\/jats:p>","DOI":"10.1002\/sim.5328","type":"journal-article","created":{"date-parts":[[2012,3,13]],"date-time":"2012-03-13T11:57:08Z","timestamp":1331639828000},"page":"2577-2587","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":285,"title":["Misuse of DeLong test to compare AUCs for nested models"],"prefix":"10.1002","volume":"31","author":[{"given":"Olga V.","family":"Demler","sequence":"first","affiliation":[{"name":"Department of Biostatistics Boston University School of Public Health Crosstown  801 Massachusetts Ave Boston MA 02118 U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael J.","family":"Pencina","sequence":"additional","affiliation":[{"name":"Department of Biostatistics Boston University School of Public Health Crosstown  801 Massachusetts Ave Boston MA 02118 U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"suffix":"Sr.","given":"Ralph\u2009B.","family":"D'Agostino","sequence":"additional","affiliation":[{"name":"Department of Mathematics and Statistics Boston University  111 Cummington Street, Boston MA 02215 U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2012,3,13]]},"reference":[{"key":"e_1_2_8_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-3462-1"},{"key":"e_1_2_8_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-77244-8"},{"key":"e_1_2_8_4_1","volume-title":"The Statistical Evaluation of Medical Tests for Classification and Prediction","author":"Pepe MS","year":"2004"},{"key":"e_1_2_8_5_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1974.tb00994.x"},{"key":"e_1_2_8_6_1","doi-asserted-by":"publisher","DOI":"10.1056\/NEJM200303133481115"},{"key":"e_1_2_8_7_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2010.461"},{"key":"e_1_2_8_8_1","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMoa0804742"},{"key":"e_1_2_8_9_1","doi-asserted-by":"publisher","DOI":"10.1161\/CIRCULATIONAHA.107.699579"},{"key":"e_1_2_8_10_1","doi-asserted-by":"publisher","DOI":"10.2307\/2531595"},{"key":"e_1_2_8_11_1","unstructured":"SAS\/STAT software Version 9.1(TS1M3) of the SAS System. Copyright \u00a9 2002\u20102003 SAS Institute Inc. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. Cary NC \u2009USA."},{"key":"e_1_2_8_12_1","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMoa055373"},{"key":"e_1_2_8_13_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.297.6.611"},{"key":"e_1_2_8_14_1","doi-asserted-by":"publisher","DOI":"10.1161\/CIRCULATIONAHA.106.672402"},{"key":"e_1_2_8_15_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2009.1757"},{"key":"e_1_2_8_16_1","doi-asserted-by":"publisher","DOI":"10.1002\/sim.4196"},{"key":"e_1_2_8_17_1","doi-asserted-by":"publisher","DOI":"10.1002\/9780470316436"},{"key":"e_1_2_8_18_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1993.10476417"},{"key":"e_1_2_8_19_1","volume-title":"Multivariate analysis","author":"Mardia KV","year":"1979"},{"key":"e_1_2_8_20_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.285.19.2486"},{"key":"e_1_2_8_21_1","doi-asserted-by":"publisher","DOI":"10.1161\/01.CIR.97.18.1837"},{"key":"e_1_2_8_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/0002-8703(91)90861-B"},{"key":"e_1_2_8_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0895-4356(01)00341-9"},{"key":"e_1_2_8_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-4541-9"},{"issue":"3","key":"e_1_2_8_25_1","first-page":"969","article-title":"Bootstrap Hypothesis Testing","volume":"48","author":"Tibshirani R","year":"1992","journal-title":"Biometrics"},{"key":"e_1_2_8_26_1","doi-asserted-by":"publisher","DOI":"10.2307\/2532163"},{"issue":"13","key":"e_1_2_8_27_1","first-page":"1","article-title":"One statistical test is sufficient for assessing new predictive markers","volume":"11","author":"Vickers AJ","year":"2011","journal-title":"BMC Medical Research Methodology"},{"key":"e_1_2_8_28_1","volume-title":"U\u2010Statistics: Theory and Practice","author":"Lee AJ","year":"1990"},{"key":"e_1_2_8_29_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177729639"},{"key":"e_1_2_8_30_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177730385"},{"key":"e_1_2_8_31_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177730196"},{"key":"e_1_2_8_32_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176350274"},{"key":"e_1_2_8_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00340061"},{"key":"e_1_2_8_34_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176345787"},{"key":"e_1_2_8_35_1","doi-asserted-by":"publisher","DOI":"10.1081\/STA-200026579"},{"key":"e_1_2_8_36_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467-9876.00238"},{"key":"e_1_2_8_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1076-6332(98)80208-0"}],"container-title":["Statistics in Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fsim.5328","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fsim.5328","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/sim.5328","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,8]],"date-time":"2023-10-08T10:29:39Z","timestamp":1696760979000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/sim.5328"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,3,13]]},"references-count":36,"journal-issue":{"issue":"23","published-print":{"date-parts":[[2012,10,15]]}},"alternative-id":["10.1002\/sim.5328"],"URL":"https:\/\/doi.org\/10.1002\/sim.5328","archive":["Portico"],"relation":{},"ISSN":["0277-6715","1097-0258"],"issn-type":[{"value":"0277-6715","type":"print"},{"value":"1097-0258","type":"electronic"}],"subject":[],"published":{"date-parts":[[2012,3,13]]}}}