{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,14]],"date-time":"2026-10-14T12:19:58Z","timestamp":1791980398860,"version":"4.4.0"},"reference-count":65,"publisher":"Wiley","issue":"5","license":[{"start":{"date-parts":[[2023,2,25]],"date-time":"2023-02-25T00:00:00Z","timestamp":1677283200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R44CA254844"],"award-info":[{"award-number":["R44CA254844"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["aapm.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Medical Physics"],"published-print":{"date-parts":[[2023,5]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Background<\/jats:title><jats:p>Contouring error is one of the top failure modes in radiation treatment. Multiple efforts have been made to develop tools to automatically detect segmentation errors. Deep learning\u2010based auto\u2010segmentation (DLAS) has been used as a baseline for flagging manual segmentation errors, but those efforts are limited to using only one or two contour comparison metrics.<\/jats:p><\/jats:sec><jats:sec><jats:title>Purpose<\/jats:title><jats:p>The purpose of this research is to develop an improved contouring quality assurance system to identify and flag manual contouring errors.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods and materials<\/jats:title><jats:p>DLAS contours were used as a reference to compare with manually segmented contours. A total of 27 geometric agreement metrics were determined from the comparisons between the two segmentation approaches. Feature selection was performed to optimize the training of a machine learning classification model to identify potential contouring errors. A public dataset with 339 cases was used to train and test the classifier. Four independent classifiers were trained using five\u2010fold cross validation, and the predictions from each classifier were ensembled using soft voting. The trained model was validated on a held\u2010out testing dataset. An additional independent clinical dataset with 60 cases was used to test the generalizability of the model. Model predictions were reviewed by an expert to confirm or reject the findings.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The proposed machine learning multiple features (ML\u2010MF) approach outperformed traditional nonmachine\u2010learning\u2010based approaches that are based on only one or two geometric agreement metrics. The machine learning model achieved recall (precision) values of 0.842 (0.899), 0.762 (0.762), 0.727 (0.842), and 0.773 (0.773) for Brainstem, Parotid_L, Parotid_R, and mandible contours, respectively compared to 0.526 (0.909), 0.619 (0.765), 0.682 (0.882), 0.773 (0.568) for an approach based solely on Dice similarity coefficient values. In the external validation dataset, 66.7, 93.3, 94.1, and 58.8% of flagged cases were confirmed to have contouring errors by an expert for Brainstem, Parotid_L, Parotid_R, and mandible contours, respectively.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>The proposed ML\u2010MF approach, which includes multiple geometric agreement metrics to flag manual contouring errors, demonstrated superior performance in comparison to traditional methods. This method is easy to implement in clinical practice and can help to reduce the significant time and labor costs associated with manual segmentation and review.<\/jats:p><\/jats:sec>","DOI":"10.1002\/mp.16299","type":"journal-article","created":{"date-parts":[[2023,2,15]],"date-time":"2023-02-15T06:40:30Z","timestamp":1676443230000},"page":"2715-2732","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":26,"title":["Contouring quality assurance methodology based on multiple geometric features against deep learning auto\u2010segmentation"],"prefix":"10.1002","volume":"50","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6551-9127","authenticated-orcid":false,"given":"Jingwei","family":"Duan","sequence":"first","affiliation":[{"name":"Department of Radiation Medicine University of Kentucky  Lexington Kentucky USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark E.","family":"Bernard","sequence":"additional","affiliation":[{"name":"Department of Radiation Medicine University of Kentucky  Lexington Kentucky USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James R.","family":"Castle","sequence":"additional","affiliation":[{"name":"Carina Medical LLC  Lexington Kentucky USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xue","family":"Feng","sequence":"additional","affiliation":[{"name":"Carina Medical LLC  Lexington Kentucky USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chi","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Internal Medicine University of Kentucky  Lexington Kentucky USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark C.","family":"Kenamond","sequence":"additional","affiliation":[{"name":"Department of Radiation Medicine University of Kentucky  Lexington Kentucky USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5570-2462","authenticated-orcid":false,"given":"Quan","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Radiation Medicine University of Kentucky  Lexington Kentucky USA"},{"name":"Department of Radiation Oncology City of Hope Comprehensive Cancer Center  Duarte California USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2023,2,25]]},"reference":[{"key":"e_1_2_8_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0167-8140(02)00063-4"},{"key":"e_1_2_8_3_1","doi-asserted-by":"publisher","DOI":"10.1200\/JCO.2008.19.6386"},{"key":"e_1_2_8_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijrobp.2005.07.968"},{"key":"e_1_2_8_5_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamaoncol.2018.0039"},{"key":"e_1_2_8_6_1","doi-asserted-by":"publisher","DOI":"10.1200\/JCO.2006.07.5937"},{"key":"e_1_2_8_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijrobp.2020.11.011"},{"key":"e_1_2_8_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2014.01.019"},{"key":"e_1_2_8_9_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.14030"},{"key":"e_1_2_8_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.semradonc.2019.02.011"},{"key":"e_1_2_8_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.semradonc.2009.11.002"},{"key":"e_1_2_8_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijrobp.2017.04.023"},{"key":"e_1_2_8_13_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.12835"},{"key":"e_1_2_8_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2013.2251421"},{"key":"e_1_2_8_15_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.15458"},{"key":"e_1_2_8_16_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.13697"},{"key":"e_1_2_8_17_1","doi-asserted-by":"publisher","DOI":"10.1088\/0031-9155\/60\/13\/5199"},{"key":"e_1_2_8_18_1","doi-asserted-by":"publisher","DOI":"10.1118\/1.4906197"},{"key":"e_1_2_8_19_1","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2020.00986"},{"key":"e_1_2_8_20_1","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2020.00524"},{"key":"e_1_2_8_21_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.13814"},{"key":"e_1_2_8_22_1","unstructured":"McCarrollR YangJ CardenasC et\u00a0al.Machine learning for the prediction of physician edits to clinical autocontours in the head\u2010and\u2010neck.Paper presented at: MEDICAL PHYSICS2017."},{"key":"e_1_2_8_23_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.13466"},{"key":"e_1_2_8_24_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.13141"},{"key":"e_1_2_8_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2019.10.019"},{"key":"e_1_2_8_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2017.11.012"},{"key":"e_1_2_8_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2006.880587"},{"key":"e_1_2_8_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2021.05.003"},{"key":"e_1_2_8_29_1","doi-asserted-by":"publisher","DOI":"10.1118\/1.4813955"},{"key":"e_1_2_8_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.232073"},{"key":"e_1_2_8_31_1","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.2015151169"},{"key":"e_1_2_8_32_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejrad.2009.01.050"},{"key":"e_1_2_8_33_1","doi-asserted-by":"publisher","DOI":"10.1038\/srep13087"},{"key":"e_1_2_8_34_1","doi-asserted-by":"publisher","DOI":"10.1118\/1.4953835"},{"key":"e_1_2_8_35_1","unstructured":"FengX ChenQ.Automatic organ segmentation on head and neck CT using 3D deep convolutional neural network.Paper presented at: MEDICAL PHYSICS2018."},{"key":"e_1_2_8_36_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.13466"},{"key":"e_1_2_8_37_1","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/ab7877"},{"key":"e_1_2_8_38_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.15525"},{"key":"e_1_2_8_39_1","doi-asserted-by":"crossref","unstructured":"GhimireK ChenQ FengX.Patch\u2010based 3D UNet for head and neck tumor segmentation with an ensemble of conventional and dilated convolutions.Paper presented at: 3D Head and Neck Tumor Segmentation in PET\/CT Challenge2020.","DOI":"10.1007\/978-3-030-67194-5_9"},{"key":"e_1_2_8_40_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.12197"},{"key":"e_1_2_8_41_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-017-10371-5"},{"key":"e_1_2_8_42_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijrobp.2009.08.078"},{"key":"e_1_2_8_43_1","doi-asserted-by":"publisher","DOI":"10.1590\/S1678-77572011000500003"},{"key":"e_1_2_8_44_1","doi-asserted-by":"publisher","DOI":"10.3857\/roj.2020.00213"},{"key":"e_1_2_8_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00066-006-1477-0"},{"key":"e_1_2_8_46_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.phro.2020.06.006"},{"key":"e_1_2_8_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2017.12.013"},{"key":"e_1_2_8_48_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13014-020-01677-2"},{"key":"e_1_2_8_49_1","doi-asserted-by":"publisher","DOI":"10.1186\/1748-717X-7-32"},{"key":"e_1_2_8_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2020.07.061"},{"key":"e_1_2_8_51_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2013.04.006"},{"issue":"2","key":"e_1_2_8_52_1","first-page":"627","article-title":"Receiver operating characteristic (ROC) curve analysis for medical diagnostic test evaluation","volume":"4","author":"Hajian\u2010Tilaki K","year":"2013","journal-title":"Casp J Intern Med"},{"key":"e_1_2_8_53_1","first-page":"2825","article-title":"Scikit\u2010learn: machine learning in Python","volume":"12","author":"Pedregosa F","year":"2011","journal-title":"J Mach Learn Res"},{"key":"e_1_2_8_54_1","unstructured":"NikolovS BlackwellS ZverovitchA et\u00a0al.Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy.arXiv preprint arXiv:1809.04430.2018."},{"key":"e_1_2_8_55_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2017.01.048"},{"key":"e_1_2_8_56_1","first-page":"37","volume-title":"Data Classification: Algorithms and Applications","author":"Tang J","year":"2014"},{"key":"e_1_2_8_57_1","doi-asserted-by":"publisher","DOI":"10.1201\/9781420050264"},{"key":"e_1_2_8_58_1","unstructured":"M\u00fcllnerD.Modern hierarchical agglomerative clustering algorithms.arXiv preprint arXiv:1109.2378.2011."},{"key":"e_1_2_8_59_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-9868.2005.00503.x"},{"key":"e_1_2_8_60_1","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v033.i01"},{"key":"e_1_2_8_61_1","doi-asserted-by":"publisher","DOI":"10.1201\/9780429246593"},{"key":"e_1_2_8_62_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2009.03.002"},{"key":"e_1_2_8_63_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02295996"},{"key":"e_1_2_8_64_1","first-page":"357","volume-title":"What is an ROC Curve?","author":"Hoo ZH","year":"2017"},{"key":"e_1_2_8_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3007291"},{"key":"e_1_2_8_66_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2015.07.041"}],"container-title":["Medical Physics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/mp.16299","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/mp.16299","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/mp.16299","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,20]],"date-time":"2023-08-20T11:43:52Z","timestamp":1692531832000},"score":1,"resource":{"primary":{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/10.1002\/mp.16299"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,25]]},"references-count":65,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2023,5]]}},"alternative-id":["10.1002\/mp.16299"],"URL":"https:\/\/doi.org\/10.1002\/mp.16299","archive":["Portico"],"relation":{},"ISSN":["0094-2405","2473-4209"],"issn-type":[{"value":"0094-2405","type":"print"},{"value":"2473-4209","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,25]]},"assertion":[{"value":"2022-06-13","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-01-26","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-02-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}