{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T16:57:36Z","timestamp":1784653056787,"version":"3.55.0"},"reference-count":23,"publisher":"Wiley","issue":"11","license":[{"start":{"date-parts":[[2018,10,12]],"date-time":"2018-10-12T00:00:00Z","timestamp":1539302400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"funder":[{"DOI":"10.13039\/501100006041","name":"Innovate UK","doi-asserted-by":"publisher","award":["600277"],"award-info":[{"award-number":["600277"]}],"id":[{"id":"10.13039\/501100006041","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["aapm.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Medical Physics"],"published-print":{"date-parts":[[2018,11]]},"abstract":"<jats:sec><jats:title>Purpose<\/jats:title><jats:p>Automated techniques for estimating the contours of organs and structures in medical images have become more widespread and a variety of measures are available for assessing their quality. Quantitative measures of geometric agreement, for example, overlap with a gold\u2010standard delineation, are popular but may not predict the level of clinical acceptance for the contouring method. Therefore, surrogate measures that relate more directly to the clinical judgment of contours, and to the way they are used in routine workflows, need to be developed. The purpose of this study is to propose a method (inspired by the Turing Test) for providing contour quality measures that directly draw upon practitioners\u2019 assessments of manual and automatic contours. This approach assumes that an inability to distinguish automatically produced contours from those of clinical experts would indicate that the contours are of sufficient quality for clinical use. In turn, it is anticipated that such contours would receive less manual editing prior to being accepted for clinical use. In this study, an initial assessment of this approach is performed with radiation oncologists and therapists.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>Eight clinical observers were presented with thoracic organ\u2010at\u2010risk contours through a web interface and were asked to determine if they were automatically generated or manually delineated. The accuracy of the visual determination was assessed, and the proportion of contours for which the source was misclassified recorded. Contours of six different organs in a clinical workflow were for 20 patient cases. The time required to edit autocontours to a clinically acceptable standard was also measured, as a gold standard of clinical utility. Established quantitative measures of autocontouring performance, such as Dice similarity coefficient with respect to the original clinical contour and the misclassification rate accessed with the proposed framework, were evaluated as surrogates of the editing time measured.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The <jats:italic>mis<\/jats:italic>classification rates for each organ were: esophagus 30.0%, heart 22.9%, left lung 51.2%, right lung 58.5%, mediastinum envelope 43.9%, and spinal cord 46.8%. The time savings resulting from editing the autocontours compared to the standard clinical workflow were 12%, 25%, 43%, 77%, 46%, and 50%, respectively, for these organs. The median Dice similarity coefficients between the clinical contours and the autocontours were 0.46, 0.90, 0.98, 0.98, 0.94, and 0.86, respectively, for these organs.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>A better correspondence with time saving was observed for the misclassification rate than the quantitative contour measures explored. From this, we conclude that the inability to accurately judge the source of a contour indicates a reduced need for editing and therefore a greater time saving overall. Hence, task\u2010based assessments of contouring performance may be considered as an additional way of evaluating the clinical utility of autosegmentation methods.<\/jats:p><\/jats:sec>","DOI":"10.1002\/mp.13200","type":"journal-article","created":{"date-parts":[[2018,9,19]],"date-time":"2018-09-19T10:36:39Z","timestamp":1537353399000},"page":"5105-5115","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":83,"title":["Comparative evaluation of autocontouring in clinical practice: A practical method using the Turing test"],"prefix":"10.1002","volume":"45","author":[{"given":"Mark J.","family":"Gooding","sequence":"first","affiliation":[{"name":"Mirada Medical Ltd Oxford Centre for Innovation New Road Oxford OX1 1BY UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Annamarie J.","family":"Smith","sequence":"additional","affiliation":[{"name":"Mirada Medical Ltd Oxford Centre for Innovation New Road Oxford OX1 1BY UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maira","family":"Tariq","sequence":"additional","affiliation":[{"name":"Mirada Medical Ltd Oxford Centre for Innovation New Road Oxford OX1 1BY UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paul","family":"Aljabar","sequence":"additional","affiliation":[{"name":"Mirada Medical Ltd Oxford Centre for Innovation New Road Oxford OX1 1BY UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Devis","family":"Peressutti","sequence":"additional","affiliation":[{"name":"Mirada Medical Ltd Oxford Centre for Innovation New Road Oxford OX1 1BY UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Judith","family":"van der Stoep","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology (MAASTRO) GROW School for Oncology and Developmental Biology Maastricht University Medical Centre+ Dr Tanslaan 12 6229ET Maastricht The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bart","family":"Reymen","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology (MAASTRO) GROW School for Oncology and Developmental Biology Maastricht University Medical Centre+ Dr Tanslaan 12 6229ET Maastricht The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daisy","family":"Emans","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology (MAASTRO) GROW School for Oncology and Developmental Biology Maastricht University Medical Centre+ Dr Tanslaan 12 6229ET Maastricht The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Djoya","family":"Hattu","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology (MAASTRO) GROW School for Oncology and Developmental Biology Maastricht University Medical Centre+ Dr Tanslaan 12 6229ET Maastricht The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Judith","family":"van Loon","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology (MAASTRO) GROW School for Oncology and Developmental Biology Maastricht University Medical Centre+ Dr Tanslaan 12 6229ET Maastricht The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maud","family":"de Rooy","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology (MAASTRO) GROW School for Oncology and Developmental Biology Maastricht University Medical Centre+ Dr Tanslaan 12 6229ET Maastricht The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rinus","family":"Wanders","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology (MAASTRO) GROW School for Oncology and Developmental Biology Maastricht University Medical Centre+ Dr Tanslaan 12 6229ET Maastricht The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephanie","family":"Peeters","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology (MAASTRO) GROW School for Oncology and Developmental Biology Maastricht University Medical Centre+ Dr Tanslaan 12 6229ET Maastricht The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tim","family":"Lustberg","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology (MAASTRO) GROW School for Oncology and Developmental Biology Maastricht University Medical Centre+ Dr Tanslaan 12 6229ET Maastricht The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Johan","family":"van Soest","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology (MAASTRO) GROW School for Oncology and Developmental Biology Maastricht University Medical Centre+ Dr Tanslaan 12 6229ET Maastricht The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andre","family":"Dekker","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology (MAASTRO) GROW School for Oncology and Developmental Biology Maastricht University Medical Centre+ Dr Tanslaan 12 6229ET Maastricht The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wouter","family":"van Elmpt","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology (MAASTRO) GROW School for Oncology and Developmental Biology Maastricht University Medical Centre+ Dr Tanslaan 12 6229ET Maastricht The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2018,10,12]]},"reference":[{"key":"e_1_2_8_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2014.09.014"},{"key":"e_1_2_8_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijrobp.2008.07.001"},{"key":"e_1_2_8_4_1","doi-asserted-by":"publisher","DOI":"10.1120\/jacmp.v15i4.4468"},{"key":"e_1_2_8_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijrobp.2013.06.1892"},{"key":"e_1_2_8_6_1","doi-asserted-by":"publisher","DOI":"10.1186\/1748-717X-8-229"},{"key":"e_1_2_8_7_1","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2016.00178"},{"key":"e_1_2_8_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijrobp.2010.07.009"},{"key":"e_1_2_8_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2010.11.017"},{"key":"e_1_2_8_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0167-8140(15)30238-3"},{"key":"e_1_2_8_11_1","doi-asserted-by":"publisher","DOI":"10.1118\/1.4871620"},{"key":"e_1_2_8_12_1","doi-asserted-by":"publisher","DOI":"10.1118\/1.4813955"},{"key":"e_1_2_8_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2003.11.010"},{"key":"e_1_2_8_14_1","doi-asserted-by":"publisher","DOI":"10.1002\/jmrs.64"},{"key":"e_1_2_8_15_1","doi-asserted-by":"publisher","DOI":"10.1093\/mind\/LIX.236.433"},{"key":"e_1_2_8_16_1","unstructured":"ShieberSM.Lessons from a restricted Turing test;1994. arXiv preprint cmp\u2010lg\/9404002."},{"key":"e_1_2_8_17_1","doi-asserted-by":"publisher","DOI":"10.1093\/mind\/LXXIII.290.234"},{"key":"e_1_2_8_18_1","first-page":"9","volume":"3","author":"Harnad S","year":"1992","journal-title":"The Turing Test is not a trick: Turing indistinguishability is a scientific criterion"},{"key":"e_1_2_8_19_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1011288000451"},{"key":"e_1_2_8_20_1","doi-asserted-by":"crossref","first-page":"e30","DOI":"10.1016\/j.ejmp.2015.10.020","article-title":"The impact of the number of atlases on the performance of automatic multi\u2010atlas contouring","volume":"31","author":"Larrue A","year":"2015","journal-title":"Phys Med: Eur J Med Phys"},{"key":"e_1_2_8_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2017.11.012"},{"key":"e_1_2_8_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00330-015-3747-7"},{"key":"e_1_2_8_23_1","first-page":"618","article-title":"An automatic mass detection system in mammograms based on complex texture features","volume":"2014","author":"Tai S","year":"2014","journal-title":"IEEE J Biomed Health Inform"},{"key":"e_1_2_8_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/RBME.2013.2295804"}],"container-title":["Medical Physics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fmp.13200","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/mp.13200","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/mp.13200","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/mp.13200","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,6]],"date-time":"2023-09-06T01:44:19Z","timestamp":1693964659000},"score":1,"resource":{"primary":{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/10.1002\/mp.13200"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,10,12]]},"references-count":23,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2018,11]]}},"alternative-id":["10.1002\/mp.13200"],"URL":"https:\/\/doi.org\/10.1002\/mp.13200","archive":["Portico"],"relation":{},"ISSN":["0094-2405","2473-4209"],"issn-type":[{"value":"0094-2405","type":"print"},{"value":"2473-4209","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,10,12]]},"assertion":[{"value":"2018-05-29","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-09-10","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-10-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}