{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T23:33:28Z","timestamp":1787009608482,"version":"build-2736575974"},"reference-count":33,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2025,11,17]],"date-time":"2025-11-17T00:00:00Z","timestamp":1763337600000},"content-version":"vor","delay-in-days":16,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100005979","name":"Lustgarten Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100005979","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,11,10]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Deep learning\u2013based pancreas segmentation in CT has advanced rapidly yet remains evaluated primarily with mean overlap metrics that fail to capture robustness\u2014defined as the proportion of cases reaching human-level performance. Models performing well on mean Dice or surface metrics can still fail unpredictably across scanners or anatomies. Because early detection and quantitative biomarkers rely on consistent segmentation, robustness is critical for clinical deployment.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Purpose<\/jats:title>\n                    <jats:p>To systematically evaluate the robustness of deep learning models for pancreas segmentation relative to human readers and to investigate an active learning strategy to improve reliability.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>We retrospectively assembled 903 venous-phase CT scans from patients with presumed normal-appearing pancreases and without known pancreatic disease (2005-2023), split into 803 for training\/validation and 100 healthy test cases. Each test case had 4 independent human segmentations. Inter-reader variability on this healthy-only test set defined the empirical human distribution, providing an upper-bound estimate of robustness. We introduced a Fractional Threshold (FT) metric, measuring the proportion of model predictions exceeding the minimum human performance. Robustness was assessed across models trained from scratch, fine-tuned, or pretrained, including both normal and abnormal cases. An active learning approach identified high-uncertainty predictions for human revision. Statistical comparisons were performed using the Wilcoxon signed-rank and proportions Z-tests.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The best model, a 3-dimensional U-Net trained from scratch, achieved a Dice Similarity Coefficient (DSC) of 0.88\u2009\u00b1\u20090.04 and Normalized Surface Dice (NSD) of 0.77\u2009\u00b1\u20090.09, approaching human-level segmentation (DSC\u2009=\u20090.89\u2009\u00b1\u20090.03; NSD\u2009=\u20090.75\u2009\u00b1\u20090.07). However, FT for DSC and NSD remained lower than human performance in most cases, indicating persistent model variability. Human-in-the-loop revision of acquisition-flagged outliers increased FT to 0.99, with an average time of 1.54\u2009minutes per case, corresponding to a 23-fold workload reduction.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Automated pancreas segmentation reduces workload but remains constrained by tail-case failures. Active learning enhances model reliability, bridging the gap between artificial intelligence and human-level performance.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/radadv\/umaf040","type":"journal-article","created":{"date-parts":[[2025,11,13]],"date-time":"2025-11-13T12:59:11Z","timestamp":1763038751000},"source":"Crossref","is-referenced-by-count":3,"title":["Benchmarking robustness of automated CT pancreas segmentation: achieving human-level reliability through human-in-the-loop optimization"],"prefix":"10.1093","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9480-6539","authenticated-orcid":false,"given":"Felipe","family":"Oviedo","sequence":"first","affiliation":[{"name":"AI for Good Lab, Microsoft Corporation , Redmond, WA, 98052,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1560-9172","authenticated-orcid":false,"given":"Felipe","family":"Lopez\u2013Ramirez","sequence":"additional","affiliation":[{"name":"Department of Radiology, Johns Hopkins University , Baltimore, MD, 21287,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Florent","family":"Tixier","sequence":"additional","affiliation":[{"name":"Department of Radiology, Johns Hopkins University , Baltimore, MD, 21287,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Satomi","family":"Kawamoto","sequence":"additional","affiliation":[{"name":"Department of Radiology, Johns Hopkins University , Baltimore, MD, 21287,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1856-5184","authenticated-orcid":false,"given":"Alejandra","family":"Blanco","sequence":"additional","affiliation":[{"name":"Department of Radiology, Johns Hopkins University , Baltimore, MD, 21287,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3812-3906","authenticated-orcid":false,"given":"Rahul","family":"Dodhia","sequence":"additional","affiliation":[{"name":"AI for Good Lab, Microsoft Corporation , Redmond, WA, 98052,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4554-5672","authenticated-orcid":false,"given":"Ralph H","family":"Hruban","sequence":"additional","affiliation":[{"name":"The Sol Goldman Pancreatic Cancer Research Center, Department of Pathology, Johns Hopkins University , Baltimore, MD, 21287,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9918-1360","authenticated-orcid":false,"given":"William B","family":"Weeks","sequence":"additional","affiliation":[{"name":"AI for Good Lab, Microsoft Corporation , Redmond, WA, 98052,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Juan M","family":"Lavista Ferres","sequence":"additional","affiliation":[{"name":"AI for Good Lab, Microsoft Corporation , Redmond, WA, 98052,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elliot K","family":"Fishman","sequence":"additional","affiliation":[{"name":"Department of Radiology, Johns Hopkins University , Baltimore, MD, 21287,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9729-2756","authenticated-orcid":false,"given":"Linda C","family":"Chu","sequence":"additional","affiliation":[{"name":"Department of Radiology, Johns Hopkins University , Baltimore, MD, 21287,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,11,17]]},"reference":[{"key":"2025121319023930700_umaf040-B1","doi-asserted-by":"publisher","first-page":"108727","DOI":"10.1109\/ACCESS.2023.3320570","article-title":"Automatic segmentation of pancreas and pancreatic tumor: a review of a decade of research","volume":"11","author":"Ghorpade","year":"2023","journal-title":"IEEE Access"},{"key":"2025121319023930700_umaf040-B2","doi-asserted-by":"publisher","author":"Xia","year":"2022","DOI":"10.1101\/2022.09.24.22280071"},{"issue":"2","key":"2025121319023930700_umaf040-B3","doi-asserted-by":"publisher","first-page":"501","DOI":"10.1007\/s00261-023-04122-6","article-title":"Deep neural network-based segmentation of normal and abnormal pancreas on abdominal CT: evaluation of global and local accuracies","volume":"49","author":"Kawamoto","year":"2024","journal-title":"Abdom Radiol (NY)"},{"issue":"10","key":"2025121319023930700_umaf040-B4","doi-asserted-by":"publisher","first-page":"3559","DOI":"10.1007\/s00261-024-04358-w","article-title":"Early detection of pancreatic cancer in the era of precision medicine","volume":"49","author":"Ahmed","year":"2024","journal-title":"Abdom Radiol (NY)"},{"issue":"5","key":"2025121319023930700_umaf040-B5","doi-asserted-by":"publisher","first-page":"689","DOI":"10.1016\/j.acra.2019.08.014","article-title":"Technical and clinical factors affecting success rate of a deep learning method for pancreas segmentation on CT","volume":"27","author":"Bagheri","year":"2020","journal-title":"Acad Radiol"},{"key":"2025121319023930700_umaf040-B6","doi-asserted-by":"publisher","first-page":"103382","DOI":"10.1016\/j.media.2024.103382","article-title":"Large-scale multi-center CT and MRI segmentation of pancreas with deep learning","volume":"99","author":"Zhang","year":"2025","journal-title":"Med Image Anal"},{"issue":"1","key":"2025121319023930700_umaf040-B7","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1016\/j.diii.2023.08.002","article-title":"Accurate non-invasive grading of nonfunctional pancreatic neuroendocrine tumors with a CT derived radiomics signature","volume":"105","author":"Javed","year":"2024","journal-title":"Diagn Interv Imaging"},{"issue":"7","key":"2025121319023930700_umaf040-B8","doi-asserted-by":"publisher","first-page":"1182","DOI":"10.1016\/j.pan.2024.09.001","article-title":"Performance of explainable artificial intelligence in guiding the management of patients with a pancreatic cyst","volume":"24","author":"Lavista Ferres","year":"2024","journal-title":"Pancreatology"},{"issue":"1","key":"2025121319023930700_umaf040-B9","doi-asserted-by":"publisher","first-page":"28","DOI":"10.1016\/j.diii.2024.08.003","article-title":"Radiomics machine learning algorithm facilitates detection of small pancreatic neuroendocrine tumors on CT","volume":"106","author":"Lopez-Ramirez","year":"2025","journal-title":"Diagn Interv Imaging"},{"issue":"10","key":"2025121319023930700_umaf040-B10","doi-asserted-by":"publisher","first-page":"6695","DOI":"10.1109\/TPAMI.2021.3100536","article-title":"AbdomenCT-1K: Is Abdominal Organ Segmentation a Solved Problem?","volume":"44","author":"Ma","year":"2022","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"issue":"1","key":"2025121319023930700_umaf040-B11","doi-asserted-by":"publisher","first-page":"4075","DOI":"10.1038\/s41598-022-07848-3","article-title":"Automated pancreas segmentation and volumetry using deep neural network on computed tomography","volume":"12","author":"Lim","year":"2022","journal-title":"Sci Rep"},{"issue":"3","key":"2025121319023930700_umaf040-B12","doi-asserted-by":"publisher","first-page":"964","DOI":"10.1007\/s00261-023-04127-1","article-title":"Assessing the robustness of a machine-learning model for early detection of pancreatic adenocarcinoma (PDA): evaluating resilience to variations in image acquisition and radiomics workflow using image perturbation methods","volume":"49","author":"Mukherjee","year":"2024","journal-title":"Abdom Radiol (NY)"},{"key":"2025121319023930700_umaf040-B13","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1016\/j.yacr.2023.05.001","article-title":"Deep learning algorithms for pancreas segmentation from radiology scans: a review","volume-title":"Adv Clin Radiol.","author":"Zhang","year":"2023"},{"issue":"5","key":"2025121319023930700_umaf040-B14","doi-asserted-by":"publisher","first-page":"e230024","DOI":"10.1148\/ryai.230024","article-title":"TotalSegmentator: robust segmentation of 104 anatomic structures in CT images","volume":"5","author":"Wasserthal","year":"2023","journal-title":"Radiol Artif Intell"},{"key":"2025121319023930700_umaf040-B15","unstructured":"Alves N, Schuurmans M, Rutkowski D, et al \u00a0The PANORAMA study protocol: pancreatic cancer diagnosis\u2014radiologists meet AI. Zenodo. 2024. 10.5281\/zenodo.10599559"},{"key":"2025121319023930700_umaf040-B16","first-page":"36620","article-title":"Abdomenatlas-8k: Annotating 8,000 ct volumes for multi-organ segmentation in three weeks","volume":"36","author":"Qu","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"5","key":"2025121319023930700_umaf040-B17","doi-asserted-by":"publisher","first-page":"1001","DOI":"10.1016\/j.pan.2021.03.016","article-title":"Quality gaps in public pancreas imaging datasets: implications & challenges for AI applications","volume":"21","author":"Suman","year":"2021","journal-title":"Pancreatology"},{"issue":"1","key":"2025121319023930700_umaf040-B18","doi-asserted-by":"publisher","first-page":"358","DOI":"10.1053\/j.gastro.2020.03.088","article-title":"AGA clinical practice update on pancreas cancer screening in high-risk individuals: expert review","volume":"159","author":"Aslanian","year":"2020","journal-title":"Gastroenterology"},{"issue":"1","key":"2025121319023930700_umaf040-B19","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1136\/gutjnl-2019-319352","article-title":"Management of patients with increased risk for familial pancreatic cancer: updated recommendations from the International Cancer of the Pancreas Screening (CAPS) Consortium","volume":"69","author":"Goggins","year":"2020","journal-title":"Gut"},{"issue":"2","key":"2025121319023930700_umaf040-B20","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1038\/s41592-020-01008-z","article-title":"Automated design of deep learning methods for biomedical image segmentation","volume":"18","author":"Isensee","year":"2021","journal-title":"Nat Methods"},{"key":"2025121319023930700_umaf040-B21","doi-asserted-by":"publisher","author":"Isensee","year":"2018","DOI":"10.48550\/arXiv.1809.10486"},{"key":"2025121319023930700_umaf040-B22","doi-asserted-by":"publisher","author":"Hatamizadeh","year":"2021","DOI":"10.48550\/arXiv.2103.10504"},{"key":"2025121319023930700_umaf040-B23","doi-asserted-by":"publisher","first-page":"272","author":"Hatamizadeh","DOI":"10.48550\/arXiv.21026"},{"key":"2025121319023930700_umaf040-B24","doi-asserted-by":"publisher","first-page":"311","DOI":"10.1007\/978-3-030-11726-9_28","volume-title":"Brainlesion Glioma Mult Scler Stroke Trauma Brain Inj","author":"Myronenko","year":"2019"},{"key":"2025121319023930700_umaf040-B25","doi-asserted-by":"publisher","author":"Li W","DOI":"10.48550\/arXiv.2501.11253"},{"key":"2025121319023930700_umaf040-B26"},{"key":"2025121319023930700_umaf040-B27","doi-asserted-by":"publisher","author":"Cardoso","year":"2022","DOI":"10.48550\/arXiv.2211.02701"},{"issue":"2","key":"2025121319023930700_umaf040-B28","doi-asserted-by":"publisher","first-page":"178","DOI":"10.1016\/S1076-6332(03)00671-8","article-title":"Statistical validation of image segmentation quality based on a spatial overlap index1: scientific reports","volume":"11","author":"Zou","year":"2004","journal-title":"Acad Radiol"},{"key":"2025121319023930700_umaf040-B29","doi-asserted-by":"publisher","author":"Nikolov","year":"2021","DOI":"10.48550\/arXiv.1809.04430"},{"issue":"1","key":"2025121319023930700_umaf040-B30","doi-asserted-by":"publisher","first-page":"6086","DOI":"10.1038\/s41598-024-56706-x","article-title":"Evaluation metrics and statistical tests for machine learning","volume":"14","author":"Rainio","year":"2024","journal-title":"Sci Rep"},{"key":"2025121319023930700_umaf040-B31","doi-asserted-by":"publisher","first-page":"399","DOI":"10.1007\/978-3-319-66179-7_46","volume-title":"Med Image Comput Comput Assist Interv \u2212 MICCAI 2017","author":"Yang","year":"2017"},{"key":"2025121319023930700_umaf040-B32","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1007\/978-3-031-17027-0_2","volume-title":"Data Augment Label Imperfections","author":"Diaz-Pinto","year":"2022"},{"key":"2025121319023930700_umaf040-B33","doi-asserted-by":"publisher","first-page":"15184","DOI":"10.48550\/arXiv.2411.03670"}],"container-title":["Radiology Advances"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/radadv\/advance-article-pdf\/doi\/10.1093\/radadv\/umaf040\/65356908\/umaf040.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/radadv\/article-pdf\/2\/6\/umaf040\/65356908\/umaf040.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/radadv\/article-pdf\/2\/6\/umaf040\/65356908\/umaf040.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,14]],"date-time":"2025-12-14T00:02:50Z","timestamp":1765670570000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/radadv\/article\/doi\/10.1093\/radadv\/umaf040\/8325238"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11]]},"references-count":33,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2025,11,10]]}},"URL":"https:\/\/doi.org\/10.1093\/radadv\/umaf040","relation":{},"ISSN":["2976-9337"],"issn-type":[{"value":"2976-9337","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,11]]},"published":{"date-parts":[[2025,11]]},"article-number":"umaf040"}}