{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,14]],"date-time":"2026-10-14T20:32:23Z","timestamp":1792009943213,"version":"4.4.0"},"reference-count":36,"publisher":"Wiley","issue":"4","license":[{"start":{"date-parts":[[2023,11,28]],"date-time":"2023-11-28T00:00:00Z","timestamp":1701129600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"content-domain":{"domain":["aapm.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Medical Physics"],"published-print":{"date-parts":[[2024,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Background<\/jats:title><jats:p>For autosegmentation models, the data used to train the model (e.g., public datasets and\/or vendor\u2010collected data) and the data on which the model is deployed in the clinic are typically not the same, potentially impacting the performance of these models by a process called domain shift. Tools to routinely monitor and predict segmentation performance are needed for quality assurance. Here, we develop an approach to perform such monitoring and performance prediction for cardiac substructure segmentation.<\/jats:p><\/jats:sec><jats:sec><jats:title>Purpose<\/jats:title><jats:p>To develop a quality assurance (QA) framework for routine or continuous monitoring of domain shift and the performance of cardiac substructure autosegmentation algorithms.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>A benchmark dataset consisting of computed tomography (CT) images along with manual cardiac substructure delineations of 241 breast cancer radiotherapy patients were collected, including one \u201cnormal\u201d image domain of clean images and five \u201cabnormal\u201d domains containing images with artifact (metal, contrast), pathology, or quality variations due to scanner protocol differences (field of view, noise, reconstruction kernel, and slice thickness). The QA framework consisted of an image domain shift detector which operated on the input CT images and a shape quality detector on the output of an autosegmentation model, and a regression model for predicting autosegmentation model performance. The image domain shift detector was composed of a trained denoising autoencoder (DAE) and two hand\u2010engineered image quality features to detect normal versus abnormal domains in the input CT images. The shape quality detector was a variational autoencoder (VAE) trained to estimate the shape quality of the auto\u2010segmentation results. The output from the image domain shift and shape quality detectors was used to train a regression model to predict the per\u2010patient segmentation accuracy, measured by Dice coefficient similarity (DSC) to physician contours. Different regression techniques were investigated including linear regression, Bagging, Gaussian process regression, random forest, and gradient boost regression. Of the 241 patients, 60 were used to train the autosegmentation models, 120 for training the QA framework, and the remaining 61 for testing the QA framework. A total of 19 autosegmentation models were used to evaluate QA framework performance, including 18 convolutional neural network (CNN)\u2010based and one transformer\u2010based model.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>When tested on the benchmark dataset, all abnormal domains resulted in a significant DSC decrease relative to the normal domain for CNN models (), but only for some domains for the transformer model. No significant relationship was found between the performance of an autosegmentation model and scanner protocol parameters () except noise (). CNN\u2010based autosegmentation models demonstrated a decreased DSC ranging from 0.07 to 0.41 with added noise, while the transformer\u2010based model was not significantly affected (ANOVA, ). For the QA framework, linear regression models with bootstrap aggregation resulted in the highest mean absolute error (MAE) of , in predicted DSC (relative to true DSC between autosegmentation and physician). MAE was lowest when combining both input (image) detectors and output (shape) detectors compared to output detectors alone.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>A QA framework was able to predict cardiac substructure autosegmentation model performance for clinically anticipated \u201cabnormal\u201d domain shifts.<\/jats:p><\/jats:sec>","DOI":"10.1002\/mp.16846","type":"journal-article","created":{"date-parts":[[2023,11,28]],"date-time":"2023-11-28T19:01:57Z","timestamp":1701198117000},"page":"2741-2758","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["A quality assurance framework for routine monitoring of deep learning cardiac substructure computed tomography segmentation models in radiotherapy"],"prefix":"10.1002","volume":"51","author":[{"given":"Xiyao","family":"Jin","sequence":"first","affiliation":[{"name":"Department of Radiation Oncology Washington University in St. Louis School of Medicine  St. Louis Missouri USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yao","family":"Hao","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Washington University in St. Louis School of Medicine  St. Louis Missouri USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jessica","family":"Hilliard","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Washington University in St. Louis School of Medicine  St. Louis Missouri USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhehao","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Washington University in St. Louis School of Medicine  St. Louis Missouri USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maria A.","family":"Thomas","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Washington University in St. Louis School of Medicine  St. Louis Missouri USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hua","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Washington University in St. Louis School of Medicine  St. Louis Missouri USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Abhinav K.","family":"Jha","sequence":"additional","affiliation":[{"name":"Department of Biomedical Engineering Washington University in St. Louis  St. Louis Missouri USA"},{"name":"Mallinckrodt Institute of Radiology Washington University in St. Louis  St. Louis Missouri USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Geoffrey D.","family":"Hugo","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Washington University in St. Louis School of Medicine  St. Louis Missouri USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2023,11,28]]},"reference":[{"key":"e_1_2_10_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2020.101714"},{"key":"e_1_2_10_3_1","doi-asserted-by":"publisher","DOI":"10.1088\/1361\u20106560\/ab7877"},{"key":"e_1_2_10_4_1","doi-asserted-by":"crossref","unstructured":"FullPM IsenseeF J\u00e4gerPF Maier\u2010HeinK.Studying robustness of semantic segmentation under domain shift in cardiac MRI. In:Lecture Notes in Computer Science (Including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2020;12592 LNCS.2021:238\u2010249. Accessed April 5.http:\/\/arxiv.org\/abs\/2011.07592","DOI":"10.1007\/978-3-030-68107-4_24"},{"key":"e_1_2_10_5_1","article-title":"Machine learning with multi\u2010site imaging data: an empirical study on the impact of scanner effects","author":"Glocker B","journal-title":"ArXiv"},{"key":"e_1_2_10_6_1","doi-asserted-by":"crossref","unstructured":"KaraniN ChaitanyaK BaumgartnerC KonukogluE.A lifelong learning approach to brain MR segmentation across scanners and protocols. In:Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol 11070 LNCS.Springer Verlag;2018:476\u2010484. doi:10.1007\/978\u20103\u2010030\u201000928\u20101_54","DOI":"10.1007\/978-3-030-00928-1_54"},{"key":"e_1_2_10_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2020.101907"},{"key":"e_1_2_10_8_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1002683"},{"key":"e_1_2_10_9_1","doi-asserted-by":"crossref","unstructured":"HuangY W\u00fcrflT BreiningerK LiuL LauritschG MaierA.Some investigations on robustness of deep learning in limited angle tomography.Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol 11070 LNCS.Springer Verlag;2018:145\u2010153. doi:10.1007\/978\u20103\u2010030\u201000928\u20101_17","DOI":"10.1007\/978-3-030-00928-1_17"},{"key":"e_1_2_10_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/J.RADONC.2020.09.008"},{"key":"e_1_2_10_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/J.PRRO.2022.01.003"},{"key":"e_1_2_10_12_1","doi-asserted-by":"publisher","DOI":"10.1002\/MP.13552"},{"key":"e_1_2_10_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/J.SEMRADONC.2022.06.011"},{"key":"e_1_2_10_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/J.MEDIA.2022.102426"},{"key":"e_1_2_10_15_1","unstructured":"WangS TarroniG QinC et\u00a0al.Deep generative model\u2010based quality control for cardiac MRI segmentation."},{"key":"e_1_2_10_16_1","unstructured":"DeVriesT TaylorGW Leveraging uncertainty estimates for predicting segmentation quality. Published online July 2.2018. doi:10.48550\/arxiv.1807.00502"},{"key":"e_1_2_10_17_1","unstructured":"RobinsonR OktayO BaiW et\u00a0al.Real\u2010time prediction of segmentation quality.International Conference on Medical Image Computing and Computer\u2010Assisted Intervention Granada Spain 16\u201020 September 2018.Springer Verlag;201811073 LNCS:578\u2010585. doi:10.1007\/978\u20103\u2010030\u201000937\u20103_66\/COVER"},{"key":"e_1_2_10_18_1","doi-asserted-by":"publisher","DOI":"10.2967\/JNUMED.121.263239"},{"key":"e_1_2_10_19_1","unstructured":"SoinA MerkowJ LongJ et\u00a0al.Chexstray: real\u2010time multi\u2010modal data concordance for drift detection in medical imaging AI. Published online 2022. Accessed November 14 2022.https:\/\/github.com\/microsoft\/MedImaging\u2010ModelDriftMonitoring"},{"key":"e_1_2_10_20_1","unstructured":"RabanserS UnnemannSG\u00a8 LiptonZC Failing loudly: an empirical study of methods for detecting dataset shift. Accessed November 14.2022.https:\/\/tensorflow.org\/tfx\/data_validation\/get_started#checking_data_skew_and_drift"},{"key":"e_1_2_10_21_1","doi-asserted-by":"crossref","unstructured":"LiuF XiaY YangD YuilleA XuD An alarm system for segmentation algorithm based on shape model:2019 IEEE\/CVF International Conference on Computer Vision (ICCV). 2019\u2010October:10651\u201010660.IEEE;2019. doi:10.1109\/ICCV.2019.01075","DOI":"10.1109\/ICCV.2019.01075"},{"key":"e_1_2_10_22_1","doi-asserted-by":"publisher","DOI":"10.1186\/S12968\u2010019\u20100523\u2010X\/FIGURES\/7"},{"key":"e_1_2_10_23_1","doi-asserted-by":"publisher","DOI":"10.1002\/MP.15237"},{"key":"e_1_2_10_24_1","doi-asserted-by":"crossref","unstructured":"HeK ZhangX RenS SunJ Deep residual learning for image recognition:Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Vol 2016\u2010December.IEEE Computer Society;2016:770\u2010778. doi:10.1109\/CVPR.2016.90","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_10_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/J.MEDIA.2007.12.003"},{"key":"e_1_2_10_26_1","unstructured":"KingmaDP WellingM Auto\u2010encoding variational bayes.2nd International Conference on Learning Representations ICLR 2014 \u2010 Conference Track Proceedings. Published online December 20.2013. doi:10.48550\/arxiv.1312.6114"},{"key":"e_1_2_10_27_1","unstructured":"ZhouD YuZ XieE et\u00a0al.Understanding the robustness in vision transformers. Published online June 28 2022:27378\u201027394. Accessed November 29.2022.https:\/\/proceedings.mlr.press\/v162\/zhou22m.html"},{"key":"e_1_2_10_28_1","doi-asserted-by":"crossref","unstructured":"ZhangC ZhangM ZhangS et\u00a0al.Delving deep into the generalization of vision transformers under distribution shifts. Published online2022:7277\u20107286. Accessed November 29 2022.https:\/\/github.com\/Phoenix1153\/ViT","DOI":"10.1109\/CVPR52688.2022.00713"},{"key":"e_1_2_10_29_1","doi-asserted-by":"publisher","DOI":"10.48550\/arxiv.2111.05464"},{"key":"e_1_2_10_30_1","unstructured":"RonnebergerO FischerP BroxT.U\u2010net: convolutional networks for biomedical image segmentation:Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol9351.Springer Verlag;2015:234\u2010241. doi:10.1007\/978\u20103\u2010319\u201024574\u20104_28"},{"key":"e_1_2_10_31_1","first-page":"30","article-title":"Attention is all you need","author":"Vaswani A","year":"2017","journal-title":"Adv Neural Inf Process Syst"},{"key":"e_1_2_10_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2020.2972701"},{"key":"e_1_2_10_33_1","doi-asserted-by":"publisher","DOI":"10.48550\/arxiv.1711.00937"},{"key":"e_1_2_10_34_1","doi-asserted-by":"publisher","DOI":"10.3389\/FONC.2020.00524\/BIBTEX"},{"key":"e_1_2_10_35_1","article-title":"Need for objective task\u2010based evaluation of AI\u2010based segmentation methods for quantitative PET","author":"Liu Z","journal-title":"ArXiv"},{"key":"e_1_2_10_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/J.CPET.2021.06.013"},{"key":"e_1_2_10_37_1","doi-asserted-by":"publisher","DOI":"10.1117\/1.JMI.4.1.011011"}],"container-title":["Medical Physics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/mp.16846","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,4]],"date-time":"2024-04-04T15:28:08Z","timestamp":1712244488000},"score":1,"resource":{"primary":{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/10.1002\/mp.16846"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,28]]},"references-count":36,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,4]]}},"alternative-id":["10.1002\/mp.16846"],"URL":"https:\/\/doi.org\/10.1002\/mp.16846","archive":["Portico"],"relation":{},"ISSN":["0094-2405","2473-4209"],"issn-type":[{"value":"0094-2405","type":"print"},{"value":"2473-4209","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,28]]},"assertion":[{"value":"2023-05-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-10-30","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-11-28","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}