{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,22]],"date-time":"2026-05-22T23:06:21Z","timestamp":1779491181574,"version":"3.53.1"},"reference-count":55,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T00:00:00Z","timestamp":1774915200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"Canada CIFAR AI Chair"},{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"publisher","award":["RGPIN-2021-03996"],"award-info":[{"award-number":["RGPIN-2021-03996"]}],"id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Fonds de recherche du Qu\u00e9bec\u2014Nature et technologies, programme rel\u00e8ve professorale","award":["312290"],"award-info":[{"award-number":["312290"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,6,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>Artificial Intelligence models are increasingly used in health care, yet global performance metrics can mask variations in reliability across individual patients or subgroups with shared attributes, called patient profiles. This study introduces predictive performance precision analysis in medicine (MED3pa), a method that identifies when models are less reliable, allowing clinicians to better assess model limitations.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>We propose a framework that estimates predictive confidence using 3 combined approaches: individualized (IPC), aggregated (APC), and mixed predictive confidence (MPC). Individualized predictive confidence estimates confidence for each patient, APC assesses it across profiles, and MPC combines both. We evaluate our method on 4 datasets: 1 simulated, 2 public, and 1 private clinical dataset. Metrics by declaration rate curves show how performance changes when retaining only the most confident predictions, while interpretable decision trees reveal profiles with higher or lower model confidence.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We demonstrate our method in internal, temporal, and external validation settings, as well as through a clinical example. In internal validation, limiting predictions to the 93% most confident cases improved sensitivity by 14.3% and the area under the receiver operating characteristic curve by 5.1%. In the clinical example, MED3pa identified a patient profile with high misclassification risk, demonstrating its potential for safer deployment.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>By identifying low-confidence predictions, our framework improves model reliability in clinical settings. It can be integrated into decision support systems to help clinicians make more informed decisions. Confidence thresholds help balance model performance with the proportion of patients for whom predictions are considered reliable.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Better leveraging confidence in model predictions could improve reliability and trustworthiness, supporting safer and more effective use in health care.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocag034","type":"journal-article","created":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T13:13:35Z","timestamp":1774962815000},"page":"1100-1111","source":"Crossref","is-referenced-by-count":0,"title":["Predictive performance precision analysis in medicine: identification of low-confidence predictions at patient and profile levels (MED3pa I)"],"prefix":"10.1093","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-8043-2185","authenticated-orcid":false,"given":"Olivier","family":"Lefebvre","sequence":"first","affiliation":[{"name":"Department of Computer Science, Universit\u00e9 de Sherbrooke , Sherbrooke, QC J1K 2R1,","place":["Canada"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"F\u00e9lix","family":"Camirand Lemyre","sequence":"additional","affiliation":[{"name":"Department of Mathematics, Universit\u00e9 de Sherbrooke , Sherbrooke, QC J1K 2R1,","place":["Canada"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jean-Fran\u00e7ois","family":"Ethier","sequence":"additional","affiliation":[{"name":"Department of Medicine, Universit\u00e9 de Sherbrooke , Sherbrooke, QC J1H 5N4,","place":["Canada"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lyna Hiba","family":"Chikouche","sequence":"additional","affiliation":[{"name":"\u00c9cole nationale Sup\u00e9rieure d\u2019Informatique Department of Higher Studies (CS), , Alger, 16309,","place":["Algeria"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ludmila","family":"Amriou","sequence":"additional","affiliation":[{"name":"\u00c9cole nationale Sup\u00e9rieure d\u2019Informatique Department of Higher Studies (CS), , Alger, 16309,","place":["Algeria"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dan","family":"Poenaru","sequence":"additional","affiliation":[{"name":"Department of Pediatric Surgery, Montreal Children\u2019s Hospital , Montreal, QC H4A 3J1,","place":["Canada"]},{"name":"Centre for Outcomes Research and Evaluation, Research Institute of the McGill University Health Centre , Montreal, QC H4A 3J1,","place":["Canada"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7639-8172","authenticated-orcid":false,"given":"Martin","family":"Valli\u00e8res","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Universit\u00e9 de Sherbrooke , Sherbrooke, QC J1K 2R1,","place":["Canada"]},{"name":"Medical Physics Unit, Department of Oncology, McGill University , Montreal, QC H3A 0G4,","place":["Canada"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,3,31]]},"reference":[{"key":"2026052218583145200_ocag034-B1","doi-asserted-by":"crossref","first-page":"900","DOI":"10.1158\/2159-8290.CD-21-0090","article-title":"Artificial intelligence in cancer research and precision medicine","volume":"11","author":"Bhinder","year":"2021","journal-title":"Cancer Discov"},{"key":"2026052218583145200_ocag034-B2","doi-asserted-by":"crossref","first-page":"1100","DOI":"10.1002\/cac2.12215","article-title":"Artificial intelligence for assisting cancer diagnosis and treatment in the era of precision medicine","volume":"41","author":"Chen","year":"2021","journal-title":"Cancer Commun (Lond)"},{"key":"2026052218583145200_ocag034-B3","doi-asserted-by":"crossref","first-page":"2300670","DOI":"10.1080\/07853890.2023.2300670","article-title":"Impact of artificial intelligence on prognosis, shared decision-making, and precision medicine for patients with inflammatory bowel disease: a perspective and expert opinion","volume":"55","author":"Pinton","year":"2023","journal-title":"Ann Med"},{"key":"2026052218583145200_ocag034-B4","doi-asserted-by":"crossref","first-page":"2069","DOI":"10.1016\/j.dld.2024.06.008","article-title":"Precision medicine: externally validated explainable AI support tool for predicting sustainability of infliximab and vedolizumab in ulcerative colitis","volume":"56","author":"Konikoff","year":"2024","journal-title":"Dig Liver Dis"},{"key":"2026052218583145200_ocag034-B5","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1016\/S2213-2600(14)70239-5","article-title":"Mortality prediction in intensive care units with the Super ICU Learner Algorithm (SICULA): a population-based study","volume":"3","author":"Pirracchio","year":"2015","journal-title":"Lancet Respir Med"},{"key":"2026052218583145200_ocag034-B6","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1038\/s41698-024-00560-7","article-title":"Development and validation of a deep learning-based microsatellite instability predictor from prostate cancer whole-slide images","volume":"8","author":"Hu","year":"2024","journal-title":"NPJ Precis Oncol"},{"key":"2026052218583145200_ocag034-B7","doi-asserted-by":"crossref","first-page":"2686","DOI":"10.1038\/s41591-023-02540-z","article-title":"External validation of AI models in health should be replaced with recurring local validation","volume":"29","author":"Youssef","year":"2023","journal-title":"Nat Med"},{"key":"2026052218583145200_ocag034-B8","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1186\/s12916-023-02779-w","article-title":"There is no such thing as a validated prediction model","volume":"21","author":"Van Calster","year":"2023","journal-title":"BMC Med"},{"key":"2026052218583145200_ocag034-B9","doi-asserted-by":"crossref","first-page":"106288","DOI":"10.1016\/j.cmpb.2021.106288","article-title":"The importance of being external. methodological insights for the external validation of machine learning models in medicine","volume":"208","author":"Cabitza","year":"2021","journal-title":"Comput Methods Programs Biomed"},{"key":"2026052218583145200_ocag034-B10","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1186\/s41512-022-00136-8","article-title":"Targeted validation: validating clinical prediction models in their intended population and setting","volume":"6","author":"Sperrin","year":"2022","journal-title":"Diagn Progn Res"},{"key":"2026052218583145200_ocag034-B11","doi-asserted-by":"crossref","first-page":"e074820","DOI":"10.1136\/bmj-2023-074820","article-title":"Evaluation of clinical prediction models (part 2): how to undertake an external validation study","volume":"384","author":"Riley","year":"2024","journal-title":"BMJ"},{"key":"2026052218583145200_ocag034-B12","author":"Ginsberg","year":"2023"},{"key":"2026052218583145200_ocag034-B13","doi-asserted-by":"crossref","first-page":"2996","DOI":"10.1038\/s41591-023-02562-7","article-title":"Clinical AI tools must convey predictive uncertainty for each individual patient","volume":"29","author":"Banerji","year":"2023","journal-title":"Nat Med"},{"key":"2026052218583145200_ocag034-B14","doi-asserted-by":"crossref","first-page":"11654","DOI":"10.1038\/s41598-022-15245-z","article-title":"Temporal quality degradation in AI models","volume":"12","author":"Vela","year":"2022","journal-title":"Sci Rep"},{"key":"2026052218583145200_ocag034-B15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-021-00504-6","article-title":"Artificial intelligence sepsis prediction algorithm learns to say \u201cI don\u2019t know\u201d","volume":"4","author":"Shashikumar","year":"2021","journal-title":"NPJ Digit Med"},{"key":"2026052218583145200_ocag034-B16","first-page":"371","article-title":"A tutorial on conformal prediction","volume":"9","author":"Shafer","year":"2008","journal-title":"J Mach Learn Res"},{"key":"2026052218583145200_ocag034-B17","first-page":"37728","volume-title":"Advances in Neural Information Processing Systems","author":"Laghuvarapu","year":"2023"},{"key":"2026052218583145200_ocag034-B18","doi-asserted-by":"crossref","first-page":"e2204569119","DOI":"10.1073\/pnas.2204569119","article-title":"Conformal prediction under feedback covariate shift for biomolecular design","volume":"119","author":"Fannjiang","year":"2022","journal-title":"Proc Natl Acad Sci"},{"key":"2026052218583145200_ocag034-B19","first-page":"2530","volume-title":"Advances in Neural Information Processing Systems","author":"Tibshirani","year":"2019"},{"key":"2026052218583145200_ocag034-B20","first-page":"1050","volume-title":"Proceedings of the 33rd International Conference on International Conference on Machine Learning\u2014Volume 48ICML\u201916","author":"Gal","year":"2016"},{"key":"2026052218583145200_ocag034-B21","first-page":"5580","volume-title":"Proceedings of the 31st International Conference on Neural Information Processing SystemsNIPS\u201917","author":"Kendall","year":"2017"},{"key":"2026052218583145200_ocag034-B22","author":"Malinin","year":"2021"},{"key":"2026052218583145200_ocag034-B23","first-page":"1613","volume-title":"Proceedings of the 32nd International Conference on Machine Learning","author":"Blundell","year":"2015"},{"key":"2026052218583145200_ocag034-B24","first-page":"1184","volume-title":"Proceedings of the 35th International Conference on Machine Learning","author":"Depeweg","year":"2018"},{"key":"2026052218583145200_ocag034-B25","doi-asserted-by":"crossref","first-page":"939","DOI":"10.1007\/s10994-022-06248-y","article-title":"Responsible model deployment via model-agnostic uncertainty learning","volume":"112","author":"Lahoti","year":"2023","journal-title":"Mach Learn"},{"key":"2026052218583145200_ocag034-B26","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1186\/s13321-023-00790-0","article-title":"Uncertain of uncertainties? A comparison of uncertainty quantification metrics for chemical data sets","volume":"15","author":"Rasmussen","year":"2023","journal-title":"J Cheminform"},{"key":"2026052218583145200_ocag034-B27","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1016\/j.patrec.2024.09.011","article-title":"Uncertainty quantification metrics for deep regression","volume":"186","author":"Lind","year":"2024","journal-title":"Pattern Recognit Lett"},{"key":"2026052218583145200_ocag034-B28","first-page":"1508","author":"Valdenegro-Toro","year":"2022"},{"key":"2026052218583145200_ocag034-B29","volume-title":"Advances in Neural Information Processing Systems","author":"Ovadia","year":"2019"},{"key":"2026052218583145200_ocag034-B30","first-page":"3566","author":"Zhang","year":"2014"},{"key":"2026052218583145200_ocag034-B31","first-page":"1467","volume-title":"Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics","author":"Chen","year":"2019"},{"key":"2026052218583145200_ocag034-B32","first-page":"1","author":"Zhou","year":"2022"},{"key":"2026052218583145200_ocag034-B33","first-page":"23660","volume-title":"Advances in Neural Information Processing Systems","author":"Seedat","year":"2022"},{"key":"2026052218583145200_ocag034-B34","author":"K\u00f6nig","year":"2020"},{"key":"2026052218583145200_ocag034-B35","first-page":"65","volume-title":"Proceedings of the Third International Workshop on Machine Learning in Systems Biology","author":"Nadeem","year":"2009"},{"key":"2026052218583145200_ocag034-B36","first-page":"212","article-title":"Derivation and experimental performance of standard and novel uncertainty calibration techniques","volume":"2024","author":"Brown","year":"2024","journal-title":"AMIA Annu Symp Proc"},{"key":"2026052218583145200_ocag034-B37","doi-asserted-by":"crossref","first-page":"334","DOI":"10.1038\/s41746-024-01275-6","article-title":"A data-driven framework for identifying patient subgroups on which an AI\/machine learning model may underperform","volume":"7","author":"Subbaswamy","year":"2024","journal-title":"NPJ Digit Med"},{"key":"2026052218583145200_ocag034-B38","doi-asserted-by":"crossref","first-page":"17816","DOI":"10.1038\/s41598-017-17876-z","article-title":"Leveraging uncertainty information from deep neural networks for disease detection","volume":"7","author":"Leibig","year":"2017","journal-title":"Sci Rep"},{"key":"2026052218583145200_ocag034-B39","first-page":"21633","volume-title":"Proceedings of the 39th International Conference on Machine Learning","author":"Tosh","year":"2022"},{"key":"2026052218583145200_ocag034-B40","doi-asserted-by":"crossref","first-page":"e053024","DOI":"10.1136\/bmjopen-2021-053024","article-title":"Do comprehensive deep learning algorithms suffer from hidden stratification? A retrospective study on pneumothorax detection in chest radiography","volume":"11","author":"Seah","year":"2021","journal-title":"BMJ Open"},{"key":"2026052218583145200_ocag034-B41","first-page":"19339","volume-title":"Advances in Neural Information Processing Systems","author":"Sohoni","year":"2020"},{"key":"2026052218583145200_ocag034-B42","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1145\/3368555.3384468","volume-title":"Proceedings of the ACM Conference on Health, Inference, and LearningCHIL \u201920","author":"Oakden-Rayner","year":"2020"},{"key":"2026052218583145200_ocag034-B43","doi-asserted-by":"crossref","first-page":"1962","DOI":"10.1145\/3531146.3533240","volume-title":"Proceedings of the 2022 ACM Conference on Fairness, Accountability, and TransparencyFAccT \u201922","author":"d\u2019Eon","year":"2022"},{"key":"2026052218583145200_ocag034-B44","doi-asserted-by":"crossref","first-page":"366","DOI":"10.1007\/978-3-319-10599-4_24","volume-title":"Computer Vision\u2014ECCV 2014","author":"Bansal","year":"2014"},{"key":"2026052218583145200_ocag034-B45","first-page":"594","volume-title":"Medical Image Computing and Computer Assisted Intervention\u2014MICCAI 2025","author":"Bissoto","year":"2025"},{"key":"2026052218583145200_ocag034-B46","first-page":"854","article-title":"A QUEST for model assessment: identifying difficult subgroups via epistemic uncertainty quantification","volume":"2023","author":"Brown","year":"2023","journal-title":"AMIA Ann Sympos Proc"},{"key":"2026052218583145200_ocag034-B47","doi-asserted-by":"crossref","first-page":"2957","DOI":"10.1001\/jama.1993.03510240069035","article-title":"A new simplified acute physiology score (SAPS II) based on a European\/North American multicenter study","volume":"270","author":"Le Gall","year":"1993","journal-title":"JAMA"},{"key":"2026052218583145200_ocag034-B48","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1038\/s41597-023-02136-9","article-title":"MIMIC-IV, a freely accessible electronic health record dataset","volume":"10","author":"Johnson","year":"2023","journal-title":"Sci Data"},{"key":"2026052218583145200_ocag034-B49","doi-asserted-by":"crossref","first-page":"180178","DOI":"10.1038\/sdata.2018.178","article-title":"The eICU Collaborative Research Database, a freely available multi-center database for critical care research","volume":"5","author":"Pollard","year":"2018","journal-title":"Sci Data"},{"key":"2026052218583145200_ocag034-B50","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1186\/s13040-021-00276-5","article-title":"Development and validation of a novel blending machine learning model for hospital mortality prediction in ICU patients with Sepsis","volume":"14","author":"Zeng","year":"2021","journal-title":"BioData Min"},{"key":"2026052218583145200_ocag034-B51","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1002\/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3","article-title":"Index for rating diagnostic tests","volume":"3","author":"Youden","year":"1950","journal-title":"Cancer"},{"key":"2026052218583145200_ocag034-B52","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1007\/s13755-024-00332-4","article-title":"Leveraging patients\u2019 longitudinal data to improve the hospital one-year mortality risk","volume":"13","author":"Laribi","year":"2025","journal-title":"Health Inf Sci Syst"},{"key":"2026052218583145200_ocag034-B53","volume-title":"Synthetic Data for Accessible Learning in Healthcare: Improving Mortality Prediction with Longitudinal Data","author":"Laribi","year":"2024"},{"key":"2026052218583145200_ocag034-B54","doi-asserted-by":"crossref","first-page":"e081554","DOI":"10.1136\/bmj-2024-081554","article-title":"FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare","volume":"388","author":"Lekadir","year":"2025","journal-title":"BMJ"},{"key":"2026052218583145200_ocag034-B55","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1038\/s41746-023-00771-5","article-title":"Patient-centric synthetic data generation, no reason to risk re-identification in biomedical data analysis","volume":"6","author":"Guillaudeux","year":"2023","journal-title":"NPJ Digit Med"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocag034\/67692597\/ocag034.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/6\/1100\/67692597\/ocag034.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/6\/1100\/67692597\/ocag034.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,22]],"date-time":"2026-05-22T22:58:44Z","timestamp":1779490724000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/33\/6\/1100\/8566228"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,31]]},"references-count":55,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2026,3,31]]},"published-print":{"date-parts":[[2026,6,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocag034","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,6]]},"published":{"date-parts":[[2026,3,31]]}}}