{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T16:49:29Z","timestamp":1777654169714,"version":"3.51.4"},"reference-count":56,"publisher":"Springer Science and Business Media LLC","issue":"8","license":[{"start":{"date-parts":[[2021,12,9]],"date-time":"2021-12-09T00:00:00Z","timestamp":1639008000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,12,9]],"date-time":"2021-12-09T00:00:00Z","timestamp":1639008000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100007316","name":"Klaus Tschira Stiftung","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100007316","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["257899354"],"award-info":[{"award-number":["257899354"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Helmholtz Association","award":["SIMCARD"],"award-info":[{"award-number":["SIMCARD"]}]},{"name":"Karlsruher Institut f\u00fcr Technologie (KIT)"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2022,8]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Throughout science and technology, receiver operating characteristic (ROC) curves and associated area under the curve (<jats:inline-formula><jats:alternatives><jats:tex-math>$$\\mathrm{AUC}$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:mi>AUC<\/mml:mi><\/mml:math><\/jats:alternatives><\/jats:inline-formula>) measures constitute powerful tools for assessing the predictive abilities of features, markers and tests in binary classification problems. Despite its immense popularity, ROC analysis has been subject to a fundamental restriction, in that it applies to dichotomous (yes or no) outcomes only. Here we introduce ROC movies and universal ROC (UROC) curves that apply to just any linearly ordered outcome, along with an associated coefficient of predictive ability (<jats:inline-formula><jats:alternatives><jats:tex-math>$${\\mathrm{CPA}}$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:mi>CPA<\/mml:mi><\/mml:math><\/jats:alternatives><\/jats:inline-formula>) measure.<jats:inline-formula><jats:alternatives><jats:tex-math>$${\\mathrm{CPA}}$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:mi>CPA<\/mml:mi><\/mml:math><\/jats:alternatives><\/jats:inline-formula>equals the area under the UROC curve, and admits appealing interpretations in terms of probabilities and rank based covariances. For binary outcomes<jats:inline-formula><jats:alternatives><jats:tex-math>$${\\mathrm{CPA}}$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:mi>CPA<\/mml:mi><\/mml:math><\/jats:alternatives><\/jats:inline-formula>equals<jats:inline-formula><jats:alternatives><jats:tex-math>$$\\mathrm{AUC}$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:mi>AUC<\/mml:mi><\/mml:math><\/jats:alternatives><\/jats:inline-formula>, and for pairwise distinct outcomes<jats:inline-formula><jats:alternatives><jats:tex-math>$${\\mathrm{CPA}}$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:mi>CPA<\/mml:mi><\/mml:math><\/jats:alternatives><\/jats:inline-formula>relates linearly to Spearman\u2019s coefficient, in the same way that the C index relates linearly to Kendall\u2019s coefficient. ROC movies, UROC curves, and<jats:inline-formula><jats:alternatives><jats:tex-math>$${\\mathrm{CPA}}$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:mi>CPA<\/mml:mi><\/mml:math><\/jats:alternatives><\/jats:inline-formula>nest and generalize the tools of classical ROC analysis, and are bound to supersede them in a wealth of applications. Their usage is illustrated in data examples from biomedicine and meteorology, where rank based measures yield new insights in the WeatherBench comparison of the predictive performance of convolutional neural networks and physical-numerical models for weather prediction.<\/jats:p>","DOI":"10.1007\/s10994-021-06114-3","type":"journal-article","created":{"date-parts":[[2021,12,9]],"date-time":"2021-12-09T21:36:16Z","timestamp":1639085776000},"page":"2769-2797","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Receiver operating characteristic (ROC) movies, universal ROC (UROC) curves, and coefficient of predictive ability (CPA)"],"prefix":"10.1007","volume":"111","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9397-3271","authenticated-orcid":false,"given":"Tilmann","family":"Gneiting","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eva-Maria","family":"Walz","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,12,9]]},"reference":[{"key":"6114_CR1","doi-asserted-by":"publisher","first-page":"1139","DOI":"10.1016\/S0031-3203(98)00154-X","volume":"32","author":"NM Adams","year":"1999","unstructured":"Adams, N. M., & Hands, D. J. (1999). Comparing classifiers when the misallocation costs are uncertain. Pattern Recognition, 32, 1139\u20131147.","journal-title":"Pattern Recognition"},{"key":"6114_CR2","doi-asserted-by":"publisher","first-page":"342","DOI":"10.1126\/science.aav7274","volume":"363","author":"RB Alley","year":"2019","unstructured":"Alley, R. B., Emanuel, K. A., & Zhang, F. (2019). Advances in weather prediction. Science, 363, 342\u2013344.","journal-title":"Science"},{"key":"6114_CR3","doi-asserted-by":"publisher","first-page":"1080","DOI":"10.1136\/bmj.332.7549.1080","volume":"332","author":"DG Altman","year":"2006","unstructured":"Altman, D. G., & Royston, P. (2006). The cost of dichotomising continuous variables. British Medical Journal, 332, 1080.","journal-title":"British Medical Journal"},{"key":"6114_CR4","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1038\/nature14956","volume":"525","author":"P Bauer","year":"2015","unstructured":"Bauer, P., Thorpe, A., & Brunet, G. (2015). The quiet revolution of numerical weather prediction. Nature, 525, 47\u201355.","journal-title":"Nature"},{"key":"6114_CR5","doi-asserted-by":"publisher","first-page":"1741","DOI":"10.1002\/qj.3523","volume":"145","author":"Z Ben Bouall\u00e8gue","year":"2019","unstructured":"Ben Bouall\u00e8gue, Z., Magnusson, L., Haiden, T., & Richardson, D. S. (2019). Monitoring trends in ensemble forecast performance focusing on surface variables and high-impact events. Quarterly Journal of the Royal Meteorological Society, 145, 1741\u20131755.","journal-title":"Quarterly Journal of the Royal Meteorological Society"},{"key":"6114_CR6","doi-asserted-by":"publisher","first-page":"3415","DOI":"10.1002\/qj.2624","volume":"141","author":"Z Ben Bouall\u00e8gue","year":"2015","unstructured":"Ben Bouall\u00e8gue, Z., Pinson, P., & Friederichs, P. (2015). Quantile forecast discrimination and value. Quarterly Journal of the Royal Meteorological Society, 141, 3415\u20133424.","journal-title":"Quarterly Journal of the Royal Meteorological Society"},{"key":"6114_CR7","unstructured":"Bi, J., & Bennett, K. P. (2003). Regression error characteristic curves. In Proceedings of the Twentieth International Conference on Machine Learning (ICML 2003) (AAAI Press)."},{"key":"6114_CR8","doi-asserted-by":"publisher","first-page":"687","DOI":"10.1002\/bimj.201200045","volume":"55","author":"P Blanche","year":"2013","unstructured":"Blanche, P., Dartigues, J.-F., & Jacqmin-Gatta, H. (2013). Review and comparison of ROC curve estimators for a time-dependent outcome with marker-dependent censoring. Biometrics Journal, 55, 687\u2013704.","journal-title":"Biometrics Journal"},{"key":"6114_CR9","doi-asserted-by":"publisher","first-page":"1145","DOI":"10.1016\/S0031-3203(96)00142-2","volume":"30","author":"AP Bradley","year":"1997","unstructured":"Bradley, A. P. (1997). The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30, 1145\u20131159.","journal-title":"Pattern Recognition"},{"key":"6114_CR10","doi-asserted-by":"crossref","unstructured":"Cap\u00e9ra\u00e0, P., & Genest, C. (1993). Spearman\u2019s $$\\rho$$ is larger than Kendall\u2019s $$\\tau$$ for positively dependent random variables. Nonparametric Statistics, 2, 183\u2013194.","DOI":"10.1080\/10485259308832551"},{"key":"6114_CR11","doi-asserted-by":"crossref","unstructured":"Christensen, D. (2005). Fast algorithms for the calculation of Kendall\u2019s $$\\tau$$. Computational Statistics, 20, 51\u201362.","DOI":"10.1007\/BF02736122"},{"key":"6114_CR12","doi-asserted-by":"crossref","unstructured":"Davison, A. C. (2003). Statistical models. Cambridge University Press.","DOI":"10.1017\/CBO9780511815850"},{"key":"6114_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1002\/hep.1840100102","volume":"10","author":"ER Dickson","year":"1989","unstructured":"Dickson, E. R., Grambsch, P. M., Fleming, T. R., Fischer, L. D., & Langworthy, A. (1989). Prognosis in primary biliary cirrhosis: Model for decision making. Hepatology, 10, 1\u20137.","journal-title":"Hepatology"},{"key":"6114_CR14","unstructured":"ECMWF Directorate (2012). Describing ECMWF\u2019s forecasts and forecasting system. ECMWF Newsletter, 133, 11\u201313."},{"key":"6114_CR15","doi-asserted-by":"publisher","first-page":"505","DOI":"10.1111\/rssb.12154","volume":"78","author":"W Ehm","year":"2016","unstructured":"Ehm, W., Gneiting, T., Jordan, A., & Kr\u00fcger, F. (2016). Of quantiles and expectiles: Consistent scoring functions, Choquet representations and forecast rankings (with discussion and rejoinder). Journal of the Royal Statistical Society Series B (Statistical Methodology), 78, 505\u2013562.","journal-title":"Journal of the Royal Statistical Society Series B (Statistical Methodology)"},{"key":"6114_CR16","doi-asserted-by":"publisher","first-page":"242","DOI":"10.1177\/0272989X9901900303","volume":"19","author":"R Etzioni","year":"1999","unstructured":"Etzioni, R., Pepe, M., Longton, G., Hu, C., & Goodman, G. (1999). Incorporating the time dimension in receiver operating characteristic curves: A case study of prostate cancer. Medical Decision Making, 19, 242\u2013251.","journal-title":"Medical Decision Making"},{"key":"6114_CR17","doi-asserted-by":"publisher","first-page":"861","DOI":"10.1016\/j.patrec.2005.10.010","volume":"27","author":"T Fawcett","year":"2006","unstructured":"Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27, 861\u2013874.","journal-title":"Pattern Recognition Letters"},{"key":"6114_CR18","doi-asserted-by":"crossref","unstructured":"Ferri, C., Hern\u00e1ndez-Orallo, J., & Salido, M. A. (2003). Volume under the ROC surface for multi-class problems. In Lavrac\u0306, N.\u00a0et\u00a0al. (Eds.), Proceedings of the 14th European conference on machine learning (pp.\u00a0108\u2013120). Springer.","DOI":"10.1007\/978-3-540-39857-8_12"},{"key":"6114_CR19","doi-asserted-by":"crossref","unstructured":"Flach, P.\u00a0A.\u00a0(2016). ROC analysis. In Encyclopedia of machine learning and data mining. Springer.","DOI":"10.1007\/978-1-4899-7502-7_739-1"},{"key":"6114_CR20","unstructured":"Fleming, T. R., & Harrington, D. P. (1991). Counting processes and survival analysis. Wiley."},{"key":"6114_CR21","doi-asserted-by":"publisher","unstructured":"Gneiting, T., & Vogel, P. (2021). Receiver operating characteristic (ROC) curves: Equivalences, beta model, and minimum distance estimation. Machine Learning. https:\/\/doi.org\/10.1007\/s10994-021-06115-2","DOI":"10.1007\/s10994-021-06115-2"},{"key":"6114_CR22","first-page":"1157","volume":"3","author":"I Guyon","year":"2003","unstructured":"Guyon, I., & Elisseeff, A. (2003). An introduction to variable and feature selection. Journal of Machine Learning Research, 3, 1157\u20131182.","journal-title":"Journal of Machine Learning Research"},{"key":"6114_CR23","unstructured":"Haiden, T., Janousek, M., Vitart, F., Ben Bouallegue, Z., Ferranti, L., Prates, F., & Richardson, D.\u00a0(2021). Evaluation of ECMWF forecasts, including the 2020 upgrade. https:\/\/www.ecmwf.int\/sites\/default\/files\/elibrary\/2021\/19879-evaluation-ecmwf-forecasts-including-2020-upgrade.pdf"},{"key":"6114_CR24","doi-asserted-by":"publisher","first-page":"171","DOI":"10.1023\/A:1010920819831","volume":"45","author":"DJ Hand","year":"2001","unstructured":"Hand, D. J., & Till, R. J. (2001). A simple generalization of the area under the ROC curve to multiple class classification problems. Machine Learning, 45, 171\u2013186.","journal-title":"Machine Learning"},{"key":"6114_CR25","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1148\/radiology.143.1.7063747","volume":"143","author":"JA Hanley","year":"1982","unstructured":"Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143, 29\u201336.","journal-title":"Radiology"},{"key":"6114_CR26","doi-asserted-by":"publisher","first-page":"361","DOI":"10.1002\/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4","volume":"15","author":"FE Harrell Jr","year":"1996","unstructured":"Harrell, F. E., Jr., Lee, K. L., & Mark, D. B. (1996). Tutorials in biostatistics: Multivariable prognostic models: Issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in Medicine, 15, 361\u2013387.","journal-title":"Statistics in Medicine"},{"key":"6114_CR27","doi-asserted-by":"publisher","first-page":"337","DOI":"10.1111\/j.0006-341X.2000.00337.x","volume":"56","author":"PJ Heagerty","year":"2000","unstructured":"Heagerty, P. J., Lumley, T., & Pepe, M. S. (2000). Time-dependent ROC curves for censored survival data and a diagnostic marker. Biometrics, 56, 337\u2013344.","journal-title":"Biometrics"},{"key":"6114_CR28","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1111\/j.0006-341X.2005.030814.x","volume":"61","author":"PJ Heagerty","year":"2005","unstructured":"Heagerty, P. J., & Zheng, Y. (2005). Survival model predictive accuracy and ROC curves. Biometrics, 61, 92\u2013105.","journal-title":"Biometrics"},{"key":"6114_CR29","doi-asserted-by":"crossref","unstructured":"Herbrich, R., Graepel, T., & Obermayer, K. (2000). Large margin rank boundaries for ordinal regression. In A. J. Smola, P. L. Bartlett, B. Sch\u00f6lkopf, & D. Schuurmans (Eds.), Advances in large margin classifiers (pp. 115\u2013132). MIT Press.","DOI":"10.7551\/mitpress\/1113.003.0010"},{"key":"6114_CR30","doi-asserted-by":"publisher","first-page":"3395","DOI":"10.1016\/j.patcog.2013.06.014","volume":"46","author":"J Hern\u00e1ndez-Orallo","year":"2013","unstructured":"Hern\u00e1ndez-Orallo, J. (2013). ROC curves for regression. Pattern Recognition, 46, 3395\u20133411.","journal-title":"Pattern Recognition"},{"key":"6114_CR31","first-page":"2813","volume":"13","author":"J Hern\u00e1ndez-Orallo","year":"2012","unstructured":"Hern\u00e1ndez-Orallo, J., Flach, P., & Ferri, C. (2012). A unified view of performance metrics: Translating threshold choice into expected classification. Journal of Machine Learning Research, 13, 2813\u20132869.","journal-title":"Journal of Machine Learning Research"},{"key":"6114_CR32","unstructured":"Hersbach, H., Bell, B., Berrisford, P., Biavati, G., Hor\u00e1nyi, A., Mu\u00f1oz Sabater, J., Nicolas, J., Peubey, C., Radu, R., Rozum, I., Schepers, D., Simmons, A., Soci, C., Dee, D., & Th\u00e9paut, J. -N.\u00a0(2018). ERA5 hourly data on single levels from 1979 to present. Copernicus Climate Change Service (C3S) Climate Data Store (CDS), https:\/\/doi.org\/10.24381\/cds.adbb2d47"},{"key":"6114_CR33","doi-asserted-by":"publisher","first-page":"299","DOI":"10.1109\/TKDE.2005.50","volume":"17","author":"J Huang","year":"2005","unstructured":"Huang, J., & Ling, C. X. (2005). Using AUC and accuracy in evaluating learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 17, 299\u2013310.","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"6114_CR34","doi-asserted-by":"publisher","first-page":"1121","DOI":"10.1002\/qj.3729","volume":"146","author":"A Kniffka","year":"2020","unstructured":"Kniffka, A., Knippertz, P., Fink, A. H., Benedett, A., Brooks, M. E., Hill, P. G., et al. (2020). An evaluation of operational and research weather forecasts for southern West Africa using observations from the DACCIWA field campaign in June-July 2016. Quarterly Journal of the Royal Meteorological Society, 146, 1121\u20131148.","journal-title":"Quarterly Journal of the Royal Meteorological Society"},{"key":"6114_CR35","doi-asserted-by":"crossref","unstructured":"Knight, W. R. (1966). A computer method for calculating Kendall\u2019s tau with ungrouped data. Journal of the American Statistical Association, 61, 436\u2013439.","DOI":"10.1080\/01621459.1966.10480879"},{"key":"6114_CR36","doi-asserted-by":"publisher","first-page":"814","DOI":"10.1080\/01621459.1958.10501481","volume":"53","author":"WH Kruskal","year":"1958","unstructured":"Kruskal, W. H. (1958). Ordinal measures of association. Journal of the American Statistical Association, 53, 814\u2013861.","journal-title":"Journal of the American Statistical Association"},{"key":"6114_CR37","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521, 436\u2013444.","journal-title":"Nature"},{"key":"6114_CR38","doi-asserted-by":"publisher","first-page":"331","DOI":"10.1175\/2008MWR2553.1","volume":"137","author":"SJ Mason","year":"2009","unstructured":"Mason, S. J., & Weigel, A. P. (2009). A generic forecast verification framework for administrative purposes. Monthly Weather Review, 137, 331\u2013349.","journal-title":"Monthly Weather Review"},{"key":"6114_CR39","doi-asserted-by":"publisher","first-page":"544","DOI":"10.1016\/j.jmva.2005.11.007","volume":"98","author":"J Ne\u0161lehov\u00e1","year":"2007","unstructured":"Ne\u0161lehov\u00e1, J. (2007). On rank correlation measures for non-continuous random variables. Journal of Multivariate Analysis, 98, 544\u2013567.","journal-title":"Journal of Multivariate Analysis"},{"key":"6114_CR40","doi-asserted-by":"crossref","unstructured":"Pencina, M. J., & D\u2019Agostino, R. B. (2004). Overall $$C$$ as a measure of discrimination in survival analysis: Model specific population value and confidence interval estimation. Statistics in Medicine, 22, 2109\u20132123.","DOI":"10.1002\/sim.1802"},{"key":"6114_CR41","doi-asserted-by":"crossref","unstructured":"Pepe, M. S. (2003). The statistical evaluation of medical tests for classification and prediction. Oxford University Press.","DOI":"10.1093\/oso\/9780198509844.001.0001"},{"key":"6114_CR42","unstructured":"Python Software Foundation. (2021). Python language reference. http:\/\/www.python.org"},{"key":"6114_CR43","unstructured":"R Core Team.\u00a0(2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https:\/\/www.r-project.org\/"},{"key":"6114_CR44","doi-asserted-by":"publisher","first-page":"e2020MS002203","DOI":"10.1029\/2020MS002203","volume":"12","author":"S Rasp","year":"2020","unstructured":"Rasp, S., Dueben, P. D., Scher, S., Weyn, J. A., Mouatadid, S., & Thuerey, N. (2020). WeatherBench: A benchmark dataset for data-driven weather forecasting. Journal of Advances in Modeling Earth Systems, 12, e2020MS002203.","journal-title":"Journal of Advances in Modeling Earth Systems"},{"key":"6114_CR45","doi-asserted-by":"publisher","first-page":"1518","DOI":"10.1126\/science.1205438","volume":"334","author":"DN Reshef","year":"2011","unstructured":"Reshef, D. N., Reshef, Y. A., Finucane, H. K., Grossman, S. R., McVean, G., Turnbaugh, P. J., Lander, E. S., Mitzenmacher, M., & Sabeti, P. C. (2011). Detecting novel associations in large data sets. Science, 334, 1518\u20131524.","journal-title":"Science"},{"key":"6114_CR46","doi-asserted-by":"crossref","unstructured":"Rosset, S., Perlich, C., & Zadrozny, B.\u00a0(2005). Ranking-based evaluation of regression models. In Proceedings of the fifth IEEE international conference on data mining (ICDM\u201905) (IEEE).","DOI":"10.1109\/ICDM.2005.126"},{"key":"6114_CR47","doi-asserted-by":"crossref","unstructured":"Schreyer, M. L., Paulin, R., & Trutschnig, W. (2017). On the exact region determined by Kendall\u2019s $$\\tau$$ and Spearman\u2019s $$\\rho$$. Journal of the Royal Statistical Society Series B (Statistical Methodology), 79, 613\u2013633.","DOI":"10.1111\/rssb.12181"},{"key":"6114_CR48","doi-asserted-by":"publisher","first-page":"799","DOI":"10.2307\/2090408","volume":"27","author":"RH Somers","year":"1962","unstructured":"Somers, R. H. (1962). A new asymmetric measure of association for ordinal variables. American Sociological Review, 27, 799\u2013811.","journal-title":"American Sociological Review"},{"key":"6114_CR49","doi-asserted-by":"publisher","first-page":"72","DOI":"10.2307\/1412159","volume":"15","author":"C Spearman","year":"1904","unstructured":"Spearman, C. (1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 72\u2013101.","journal-title":"American Journal of Psychology"},{"key":"6114_CR50","doi-asserted-by":"publisher","first-page":"1285","DOI":"10.1126\/science.3287615","volume":"240","author":"JA Swets","year":"1988","unstructured":"Swets, J. A. (1988). Measuring the accuracy of diagnostic systems. Science, 240, 1285\u20131293.","journal-title":"Science"},{"key":"6114_CR51","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.patrec.2007.07.019","volume":"29","author":"W Waegeman","year":"2008","unstructured":"Waegeman, W., De Bets, B., & Boullart, L. (2008). ROC analysis in ordinary regression learning. Pattern Recognition Letters, 29, 1\u20139.","journal-title":"Pattern Recognition Letters"},{"key":"6114_CR52","doi-asserted-by":"publisher","first-page":"e2020MS002109","DOI":"10.1029\/2020MS002109","volume":"12","author":"JA Weyn","year":"2020","unstructured":"Weyn, J. A., Durran, D. R., & Caruana, R. (2020). Improving data-driven global weather prediction using deep convolutional networks on a cubed sphere. Journal of Advances in Modeling Earth Systems, 12, e2020MS002109.","journal-title":"Journal of Advances in Modeling Earth Systems"},{"key":"6114_CR53","doi-asserted-by":"publisher","first-page":"547","DOI":"10.1093\/biomet\/asy021","volume":"105","author":"L Weihs","year":"2018","unstructured":"Weihs, L., Drton, M., & Meinshausen, N. (2018). Symmetric rank covariances: A generalized framework for nonparametric measures of dependence. Biometrika, 105, 547\u2013562.","journal-title":"Biometrika"},{"key":"6114_CR54","unstructured":"Wilks, D. S. (2019). Statistical methods in the atmospheric sciences (4th ed.). Elsevier."},{"key":"6114_CR55","doi-asserted-by":"publisher","first-page":"358","DOI":"10.1214\/aoms\/1177731875","volume":"11","author":"MA Woodbury","year":"1940","unstructured":"Woodbury, M. A. (1940). Rank correlation when there are equal variates. Annals of Mathematical Statistics, 11, 358\u2013362.","journal-title":"Annals of Mathematical Statistics"},{"key":"6114_CR56","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v053.i01","volume":"53","author":"Y Xie","year":"2013","unstructured":"Xie, Y. (2013). animation, an R package for creating animations and demonstrating statistical methods. Journal of Statistical Software, 53, 1\u201327.","journal-title":"Journal of Statistical Software"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-021-06114-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-021-06114-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-021-06114-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,14]],"date-time":"2024-09-14T04:11:50Z","timestamp":1726287110000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-021-06114-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,9]]},"references-count":56,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2022,8]]}},"alternative-id":["6114"],"URL":"https:\/\/doi.org\/10.1007\/s10994-021-06114-3","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,9]]},"assertion":[{"value":"4 June 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 June 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 October 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 December 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}