{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T04:02:25Z","timestamp":1791518545740,"version":"4.3.4"},"reference-count":88,"publisher":"National Academy of Sciences","issue":"34","license":[{"start":{"date-parts":[[2025,8,20]],"date-time":"2025-08-20T00:00:00Z","timestamp":1755648000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"HHS | NIH","doi-asserted-by":"publisher","award":["R21NS113016"],"award-info":[{"award-number":["R21NS113016"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"HHS | NIH","doi-asserted-by":"publisher","award":["RA37CA230400"],"award-info":[{"award-number":["RA37CA230400"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"HHS | NIH","doi-asserted-by":"publisher","award":["U01CA230691"],"award-info":[{"award-number":["U01CA230691"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Oncology Core CA","award":["06973"],"award-info":[{"award-number":["06973"]}]},{"name":"Benjamin Baker Endowment","award":["80049589"],"award-info":[{"award-number":["80049589"]}]},{"DOI":"10.13039\/100000002","name":"HHS | NIH","doi-asserted-by":"publisher","award":["80057309"],"award-info":[{"award-number":["80057309"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"HHS | NIH","doi-asserted-by":"publisher","award":["U01CA230691"],"award-info":[{"award-number":["U01CA230691"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"HHS | NIH","doi-asserted-by":"publisher","award":["5P50CA062924-22"],"award-info":[{"award-number":["5P50CA062924-22"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000925","name":"Federal Government | DHAC | National Health and Medical Research Council","doi-asserted-by":"publisher","award":["APP1194970"],"award-info":[{"award-number":["APP1194970"]}],"id":[{"id":"10.13039\/501100000925","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.pnas.org"],"crossmark-restriction":true},"short-container-title":["Proc. Natl. Acad. Sci. U.S.A."],"published-print":{"date-parts":[[2025,8,26]]},"abstract":"<jats:p>AI is now a cornerstone of modern dataset analysis. In many real world applications, practitioners are concerned with controlling specific kinds of errors, rather than minimizing the overall number of errors. For example, biomedical screening assays may primarily be concerned with mitigating the number of false positives rather than false negatives. Quantifying uncertainty in AI-based predictions, and in particular those controlling specific kinds of errors, remains theoretically and practically challenging. We develop a strategy called multidimensional informed generalized hypothesis testing (MIGHT) which we prove accurately quantifies uncertainty and confidence given sufficient data, and concomitantly controls for particular error types. Our key insight was that it is possible to integrate canonical cross-validation and parametric calibration procedures within a nonparametric ensemble method. Simulations demonstrate that while typical AI based-approaches cannot be trusted to obtain the truth, MIGHT can be. We apply MIGHT to answer an open question in liquid biopsies using circulating cell-free DNA (ccfDNA) in individuals with or without cancer: Which biomarkers, or combinations thereof, can we trust? Performance estimates produced by MIGHT on ccfDNA data have coefficients of variation that are often orders of magnitude lower than other state of the art algorithms such as support vector machines, random forests, and Transformers, while often also achieving higher sensitivity. We find that combinations of variable sets often decrease rather than increase sensitivity over the optimal single variable set because some variable sets add more noise than signal. This work demonstrates the importance of quantifying uncertainty and confidence\u2014with theoretical guarantees\u2014for the interpretation of real-world data.<\/jats:p>","DOI":"10.1073\/pnas.2424203122","type":"journal-article","created":{"date-parts":[[2025,8,20]],"date-time":"2025-08-20T14:49:18Z","timestamp":1755701358000},"update-policy":"https:\/\/doi.org\/10.1073\/pnas.cm10313","source":"Crossref","is-referenced-by-count":5,"title":["Minimizing and quantifying uncertainty in AI-informed decisions: Applications in medicine"],"prefix":"10.1073","volume":"122","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5311-2001","authenticated-orcid":false,"given":"Samuel D.","family":"Curtis","sequence":"first","affiliation":[{"name":"Department of Pharmacology and Molecular Sciences, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Department of Oncology, the Sidney Kimmel Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Ludwig Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Sol Goldman Pancreatic Cancer Research Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8455-4243","authenticated-orcid":false,"given":"Sambit","family":"Panda","sequence":"additional","affiliation":[{"name":"Department of Biomedical Engineering, Johns Hopkins University","place":["Baltimore"]},{"name":"Institute for Computational Medicine, Johns Hopkins University","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8421-365X","authenticated-orcid":false,"given":"Adam","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Columbia University","place":["New York"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Haoyin","family":"Xu","sequence":"additional","affiliation":[{"name":"Department of Biomedical Engineering, Johns Hopkins University","place":["Baltimore"]},{"name":"Institute for Computational Medicine, Johns Hopkins University","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-0689-7487","authenticated-orcid":false,"given":"Yuxin","family":"Bai","sequence":"additional","affiliation":[{"name":"Department of Biomedical Engineering, Johns Hopkins University","place":["Baltimore"]},{"name":"Institute for Computational Medicine, Johns Hopkins University","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Itsuki","family":"Ogihara","sequence":"additional","affiliation":[{"name":"Department of Biomedical Engineering, Johns Hopkins University","place":["Baltimore"]},{"name":"Institute for Computational Medicine, Johns Hopkins University","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eliza","family":"O\u2019Reilly","sequence":"additional","affiliation":[{"name":"Department of Applied Mathematics and Statistic, Johns Hopkins University","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuxuan","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Oncology, the Sidney Kimmel Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Ludwig Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Sol Goldman Pancreatic Cancer Research Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lisa","family":"Dobbyn","sequence":"additional","affiliation":[{"name":"Department of Oncology, the Sidney Kimmel Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Ludwig Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Sol Goldman Pancreatic Cancer Research Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maria","family":"Popoli","sequence":"additional","affiliation":[{"name":"Department of Oncology, the Sidney Kimmel Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Ludwig Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Sol Goldman Pancreatic Cancer Research Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Janine","family":"Ptak","sequence":"additional","affiliation":[{"name":"Department of Oncology, the Sidney Kimmel Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Ludwig Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Sol Goldman Pancreatic Cancer Research Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The HHMI, Johns Hopkins University School of Medicine","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nadine","family":"Nehme","sequence":"additional","affiliation":[{"name":"Department of Oncology, the Sidney Kimmel Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Ludwig Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Sol Goldman Pancreatic Cancer Research Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Natalie","family":"Silliman","sequence":"additional","affiliation":[{"name":"Department of Oncology, the Sidney Kimmel Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Ludwig Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Sol Goldman Pancreatic Cancer Research Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The HHMI, Johns Hopkins University School of Medicine","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeanne","family":"Tie","sequence":"additional","affiliation":[{"name":"Division of Personalized Oncology, Walter and Eliza Hall Institute of Medical Research","place":["Parkville, Australia"]},{"name":"Sir Peter MacCallum Department of Oncology, University of Melbourne","place":["Melbourne, Australia"]},{"name":"Faculty of Medicine, Dentistry and Health Sciences, University of Melbourne","place":["Melbourne, Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter","family":"Gibbs","sequence":"additional","affiliation":[{"name":"Division of Personalized Oncology, Walter and Eliza Hall Institute of Medical Research","place":["Parkville, Australia"]},{"name":"Faculty of Medicine, Dentistry and Health Sciences, University of Melbourne","place":["Melbourne, Australia"]},{"name":"Department of Medical Oncology, Western Health","place":["Melbourne, Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8382-5080","authenticated-orcid":false,"given":"Lan T.","family":"Ho-Pham","sequence":"additional","affiliation":[{"name":"BioMedical Research Center, Pham Ngoc Thach University of Medicine","place":["Ho Chi Minh City, Vietnam"]},{"name":"Clinical Genetics Research Group, Saigon Precision Medicine Research Center","place":["Ho Chi Minh City, Vietnam"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bich N. H.","family":"Tran","sequence":"additional","affiliation":[{"name":"Saigon Precision Medicine Research Center","place":["Ho Chi Minh City, Vietnam"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thach S.","family":"Tran","sequence":"additional","affiliation":[{"name":"Saigon Precision Medicine Research Center","place":["Ho Chi Minh City, Vietnam"]},{"name":"School of Biomedical Engineering, University of Technology","place":["Sydney, Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tuan V.","family":"Nguyen","sequence":"additional","affiliation":[{"name":"Saigon Precision Medicine Research Center","place":["Ho Chi Minh City, Vietnam"]},{"name":"School of Biomedical Engineering, University of Technology","place":["Sydney, Australia"]},{"name":"T\u00e2m Anh Research Institute","place":["Ho Chi Minh City, Vietnam"]},{"name":"Centre for Health Technologies, University of Technology","place":["Sydney, Australia"]},{"name":"School of Population Health, University of New South Wales","place":["Kensington, Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ehsan","family":"Irajizad","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, The University of Texas MD Anderson Cancer Center","place":["Houston"]},{"name":"Department of Clinical Cancer Prevention, The University of Texas MD Anderson Cancer Center","place":["Houston"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Goggins","sequence":"additional","affiliation":[{"name":"Department of Oncology, the Sidney Kimmel Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Sol Goldman Pancreatic Cancer Research Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Department of Pathology, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Department of Medicine, Johns Hopkins Medical Institutes","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christopher L.","family":"Wolfgang","sequence":"additional","affiliation":[{"name":"Department of Surgery, New York University Langone","place":["New York City"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tian-Li","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Pathology, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Department of Gynecology and Obstetrics, Johns Hopkins Medical Institutions","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ie-Ming","family":"Shih","sequence":"additional","affiliation":[{"name":"Department of Pathology, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Department of Gynecology and Obstetrics, Johns Hopkins Medical Institutions","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amanda","family":"Fader","sequence":"additional","affiliation":[{"name":"Department of Gynecology and Obstetrics, Johns Hopkins Medical Institutions","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anne Marie","family":"Lennon","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Pittsburgh Medical Center","place":["Pittsburgh"]},{"name":"Department of Epidemiology, University of Pittsburgh Medical Center","place":["Pittsburgh"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ralph H.","family":"Hruban","sequence":"additional","affiliation":[{"name":"Department of Oncology, the Sidney Kimmel Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Department of Pathology, Johns Hopkins University School of Medicine","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chetan","family":"Bettegowda","sequence":"additional","affiliation":[{"name":"Department of Oncology, the Sidney Kimmel Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Ludwig Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Sol Goldman Pancreatic Cancer Research Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Department of Neurosurgery, Johns Hopkins University School of Medicine","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lucy","family":"Gilbert","sequence":"additional","affiliation":[{"name":"Department of Oncology, McGill University Health Centre, Montreal","place":["Canada"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kenneth W.","family":"Kinzler","sequence":"additional","affiliation":[{"name":"Department of Oncology, the Sidney Kimmel Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Ludwig Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Sol Goldman Pancreatic Cancer Research Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nickolas","family":"Papadopoulos","sequence":"additional","affiliation":[{"name":"Department of Oncology, the Sidney Kimmel Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Ludwig Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Sol Goldman Pancreatic Cancer Research Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bert","family":"Vogelstein","sequence":"additional","affiliation":[{"name":"Department of Oncology, the Sidney Kimmel Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Ludwig Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Sol Goldman Pancreatic Cancer Research Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The HHMI, Johns Hopkins University School of Medicine","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2487-6237","authenticated-orcid":false,"given":"Joshua T.","family":"Vogelstein","sequence":"additional","affiliation":[{"name":"Department of Biomedical Engineering, Johns Hopkins University","place":["Baltimore"]},{"name":"Institute for Computational Medicine, Johns Hopkins University","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christopher","family":"Douville","sequence":"additional","affiliation":[{"name":"Department of Oncology, the Sidney Kimmel Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Ludwig Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"The Sol Goldman Pancreatic Cancer Research Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University School of Medicine","place":["Baltimore"]},{"name":"Department of Medicine, Johns Hopkins Medical Institutes","place":["Baltimore"]},{"name":"Division of Quantitative Sciences, Johns Hopkins University School of Medicine","place":["Baltimore"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"341","published-online":{"date-parts":[[2025,8,20]]},"reference":[{"key":"e_1_3_4_1_2","volume-title":"Patterns, Predictions, and Actions: Foundations of Machine Learning","author":"Hardt M.","year":"2022","unstructured":"M. Hardt, B. Recht, Patterns, Predictions, and Actions: Foundations of Machine Learning (Princeton University Press, 2022)."},{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1901326117"},{"key":"e_1_3_4_3_2","doi-asserted-by":"crossref","unstructured":"S. M. Ghasem A novel method to guide biomarker combinations to optimize the sensitivity. bioRxiv [Preprint] (2024). https:\/\/www.biorxiv.org\/content\/10.1101\/2024.04.12.589302v1 (Accessed 15 April 2024).","DOI":"10.1101\/2024.04.12.589302"},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1111\/1541-0420.00071"},{"key":"e_1_3_4_5_2","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176325633"},{"key":"e_1_3_4_6_2","doi-asserted-by":"crossref","unstructured":"A. Farago G. Lugosi Strong universal consistency of neural network classifiers. IEEE Trans. Inf. Theory 39 1146\u20131151 (1993).","DOI":"10.1109\/18.243433"},{"key":"e_1_3_4_7_2","first-page":"2015","article-title":"Consistency of random forests and other averaging classifiers","volume":"9","author":"Biau G.","year":"2008","unstructured":"G. Biau, L. Devroye, G. Lugosi, Consistency of random forests and other averaging classifiers. J. Mach. Learn. Res. 9, 2015\u20132033 (2008).","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/18.312167"},{"key":"e_1_3_4_9_2","doi-asserted-by":"crossref","unstructured":"A. Niculescu-Mizil R. Caruana \u201cPredicting good probabilities with supervised learning\u201d in Proceedings of the 22nd International Conference on Machine Learning (ACM Press 2005) pp. 625\u2013632.","DOI":"10.1145\/1102351.1102430"},{"key":"e_1_3_4_10_2","doi-asserted-by":"publisher","DOI":"10.1002\/sim.9921"},{"key":"e_1_3_4_11_2","unstructured":"C. Guo G. Pleiss Y. Sun K. Q. Weinberger On calibration of modern neural networks. arXiv [Preprint] (2017). https:\/\/arxiv.org\/abs\/1706.04599 (Accessed 8 November 2024)."},{"key":"e_1_3_4_12_2","unstructured":"J. Nixon Measuring calibration in deep learning. arXiv [Preprint] (2019). https:\/\/arxiv.org\/abs\/1904.01685 (Accessed 8 November 2024)."},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/1113.003.0008"},{"key":"e_1_3_4_14_2","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-023-02997-5"},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13104-021-05875-3"},{"key":"e_1_3_4_16_2","unstructured":"B. Recht R. Roelofs L. Schmidt V. Shankar \u201cDo imagenet classifiers generalize to imagenet?\u201d in Proceedings of the 36th International Conference on Machine Learning K. Chaudhuri R. Salakhutdinov Eds. (Proceedings of Machine Learning Research (PMLR) 2019) 09\u201315 Jun 2019 pp. 5389\u20135400."},{"key":"e_1_3_4_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2009.15"},{"key":"e_1_3_4_18_2","series-title":"Springer Series in Statistics","volume-title":"Permutation, Parametric, and Bootstrap Tests of Hypotheses","author":"Good P. I.","year":"2004","unstructured":"P. I. Good, Permutation, Parametric, and Bootstrap Tests of Hypotheses (Springer Series in Statistics, Springer, ed. 3, 2004).","edition":"3"},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2012.6247923"},{"key":"e_1_3_4_20_2","first-page":"9","article-title":"A blood-based metabolomic signature predictive of risk for pancreatic cancer","volume":"4","author":"Irajizad E.","year":"2023","unstructured":"E. Irajizad , A blood-based metabolomic signature predictive of risk for pancreatic cancer. Cell Rep. Med. 4, 9 (2023).","journal-title":"Cell Rep. Med."},{"key":"e_1_3_4_21_2","unstructured":"R. Perry mvlearn: Multiview machine learning in python. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/2005.11890 (Accessed 8 November 2024)."},{"key":"e_1_3_4_22_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.adk1213"},{"key":"e_1_3_4_23_2","volume-title":"Mathematical Statistics: Basic Ideas and Selected Topics","author":"Bickel P. J.","year":"2001","unstructured":"P. J. Bickel, K. A. Doksum, Mathematical Statistics: Basic Ideas and Selected Topics (Prentice Hall, ed. 2, 2001).","edition":"2"},{"key":"e_1_3_4_24_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.45.8.1288"},{"key":"e_1_3_4_25_2","doi-asserted-by":"publisher","DOI":"10.2307\/2530946"},{"key":"e_1_3_4_26_2","doi-asserted-by":"crossref","unstructured":"A. Blum A. Kalai J. Langford \u201cBeating the hold-out: bounds for K-fold and progressive cross-validation\u201d in Proceedings of the Twelfth Annual Conference on Computational Learning Theory (ACM 1999) pp. 203\u2013208.","DOI":"10.1145\/307400.307439"},{"key":"e_1_3_4_27_2","unstructured":"I. Ghosal Y. Zhou G. Hooker The infinitesimal jackknife and combinations of models. arXiv [Preprint] (2022). https:\/\/arxiv.org\/abs\/2209.00147 (Accessed 8 November 2024)."},{"key":"e_1_3_4_28_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1510489113"},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2017.1319839"},{"key":"e_1_3_4_30_2","doi-asserted-by":"publisher","DOI":"10.1214\/18-AOS1709"},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1804597116"},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.2016.2514489"},{"key":"e_1_3_4_33_2","first-page":"1","article-title":"Quantifying uncertainty in random forests via confidence intervals and hypothesis tests","volume":"17","author":"Mentch L.","year":"2016","unstructured":"L. Mentch, G. Hooker, Quantifying uncertainty in random forests via confidence intervals and hypothesis tests. J. Mach. Learn. Res. 17, 1\u201341 (2016).","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_4_34_2","first-page":"1","article-title":"Distributional random forests: Heterogeneity adjustment and multivariate distributional regression","volume":"23","author":"Cevid D.","year":"2022","unstructured":"D. Cevid, L. Michel, J. N\u00e4f, P. B\u00fchlmann, N. Meinshausen, Distributional random forests: Heterogeneity adjustment and multivariate distributional regression. J. Mach. Learn. Res. 23, 1\u201379 (2022).","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_4_35_2","volume-title":"The Energy of Data and Distance Correlation","author":"Szekely G. J.","year":"2023","unstructured":"G. J. Szekely, M. L. Rizzo, The Energy of Data and Distance Correlation (Chapman and Hall\/CRC, ed. 1, 2023).","edition":"1"},{"key":"e_1_3_4_36_2","first-page":"1","article-title":"Scalable and efficient hypothesis testing with random forests","volume":"23","author":"Coleman T.","year":"2022","unstructured":"T. Coleman, W. Peng, L. Mentch, Scalable and efficient hypothesis testing with random forests. J. Mach. Learn. Res. 23, 1\u201335 (2022).","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_4_37_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-021-06030-6"},{"key":"e_1_3_4_38_2","doi-asserted-by":"crossref","unstructured":"T. Hsing S. Attoor Dougherty relation between permutation-test p values and classifier error estimates. Machine Learning 52 11\u201330 (2003).","DOI":"10.1023\/A:1023985022691"},{"key":"e_1_3_4_39_2","doi-asserted-by":"publisher","DOI":"10.1214\/20-AOS1962"},{"key":"e_1_3_4_40_2","unstructured":"D. Lopez-Paz M. Oquab Revisiting classifier two-sample tests. arXiv [Preprint] (2025). https:\/\/arxiv.org\/abs\/1610.06545 (Accessed 14 October 2024)."},{"key":"e_1_3_4_41_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aaw3616"},{"key":"e_1_3_4_42_2","doi-asserted-by":"publisher","DOI":"10.3390\/diagnostics12040978"},{"key":"e_1_3_4_43_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajhg.2019.10.005"},{"key":"e_1_3_4_44_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-020-0915-3"},{"key":"e_1_3_4_45_2","doi-asserted-by":"publisher","DOI":"10.1158\/2159-8290.CD-19-0622"},{"key":"e_1_3_4_46_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tig.2023.01.004"},{"key":"e_1_3_4_47_2","doi-asserted-by":"publisher","DOI":"10.1038\/nrc.2017.7"},{"key":"e_1_3_4_48_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aar3247"},{"key":"e_1_3_4_49_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.annonc.2020.02.011"},{"key":"e_1_3_4_50_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0023418"},{"key":"e_1_3_4_51_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10555-016-9629-x"},{"key":"e_1_3_4_52_2","doi-asserted-by":"publisher","DOI":"10.3109\/08958378.2014.955932"},{"key":"e_1_3_4_53_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.xgen.2022.100242"},{"key":"e_1_3_4_54_2","doi-asserted-by":"publisher","DOI":"10.1101\/gr.242719.118"},{"key":"e_1_3_4_55_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-022-05606-z"},{"key":"e_1_3_4_56_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.abb9601"},{"key":"e_1_3_4_57_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ccell.2022.10.022"},{"key":"e_1_3_4_58_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-024-08328-6"},{"key":"e_1_3_4_59_2","doi-asserted-by":"crossref","unstructured":"F. Mitelman Cancer: Chromosomal abnormalities. eLS 1\u20139 (2017).","DOI":"10.1002\/9780470015902.a0005552.pub3"},{"key":"e_1_3_4_60_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature03099"},{"key":"e_1_3_4_61_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41576-019-0171-x"},{"key":"e_1_3_4_62_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-cancerbio-042616-072231"},{"key":"e_1_3_4_63_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1717846115"},{"key":"e_1_3_4_64_2","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gky1263"},{"key":"e_1_3_4_65_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-017-00965-y"},{"key":"e_1_3_4_66_2","doi-asserted-by":"publisher","DOI":"10.1126\/scitranslmed.abm6863"},{"key":"e_1_3_4_67_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2220982120"},{"key":"e_1_3_4_68_2","doi-asserted-by":"publisher","DOI":"10.1126\/scitranslmed.aaz3088"},{"key":"e_1_3_4_69_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cell.2015.11.050"},{"key":"e_1_3_4_70_2","doi-asserted-by":"publisher","DOI":"10.1186\/gb-2011-12-12-236"},{"key":"e_1_3_4_71_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1500076112"},{"key":"e_1_3_4_72_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pgen.1006162"},{"key":"e_1_3_4_73_2","doi-asserted-by":"publisher","DOI":"10.1373\/clinchem.2006.068577"},{"key":"e_1_3_4_74_2","doi-asserted-by":"publisher","DOI":"10.1126\/scitranslmed.aat4921"},{"key":"e_1_3_4_75_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-1272-6"},{"key":"e_1_3_4_76_2","doi-asserted-by":"publisher","DOI":"10.1126\/scitranslmed.adi3883"},{"key":"e_1_3_4_77_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11222-021-10057-z"},{"key":"e_1_3_4_78_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1910041117"},{"key":"e_1_3_4_79_2","doi-asserted-by":"publisher","DOI":"10.1038\/nrg2825"},{"key":"e_1_3_4_80_2","unstructured":"E. W. Bridgeford Batch effects are causal effects: Applications in human connectomics. bioRxiv [Preprint] (2023). https:\/\/www.biorxiv.org\/content\/10.1101\/2021.09.03.458920v4 (Accessed 8 November 2024)."},{"key":"e_1_3_4_81_2","doi-asserted-by":"publisher","DOI":"10.1038\/nmeth.1923"},{"key":"e_1_3_4_82_2","unstructured":"Picard A set of command line tools (in Java) for manipulating high-throughput sequencing (HTS) data and formats such as SAM\/BAM\/CRAM and VCF. https:\/\/broadinstitute.github.io\/picard\/. Accessed 6 June 2025."},{"key":"e_1_3_4_83_2","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btp352"},{"key":"e_1_3_4_84_2","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btq033"},{"key":"e_1_3_4_85_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-2493-4"},{"key":"e_1_3_4_86_2","unstructured":"A. F. A. Smit R. Hubley P. Green RepeatMasker Open-4.0. 2013-2015. http:\/\/www.repeatmasker.org. Accessed 6 June 2025."},{"key":"e_1_3_4_87_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-019-12954-4"},{"key":"e_1_3_4_88_2","unstructured":"S. Curtis . Minimizing and quantifying uncertainty in AI-informed decisions- applications in medicine. European Genome-Phenome Archive. https:\/\/ega-archive.org\/studies\/EGAS00001007763. Deposited 29 July 2025."}],"container-title":["Proceedings of the National Academy of Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pnas.org\/doi\/pdf\/10.1073\/pnas.2424203122","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,5]],"date-time":"2026-05-05T15:38:40Z","timestamp":1777995520000},"score":1,"resource":{"primary":{"URL":"https:\/\/pnas.org\/doi\/10.1073\/pnas.2424203122"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,20]]},"references-count":88,"journal-issue":{"issue":"34","published-print":{"date-parts":[[2025,8,26]]}},"alternative-id":["10.1073\/pnas.2424203122"],"URL":"https:\/\/doi.org\/10.1073\/pnas.2424203122","relation":{},"ISSN":["0027-8424","1091-6490"],"issn-type":[{"value":"0027-8424","type":"print"},{"value":"1091-6490","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,20]]},"assertion":[{"value":"2024-12-02","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-08","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-08-20","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e2424203122"}}