{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T14:11:19Z","timestamp":1773843079050,"version":"3.50.1"},"reference-count":12,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2022,1,26]],"date-time":"2022-01-26T00:00:00Z","timestamp":1643155200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,1,26]],"date-time":"2022-01-26T00:00:00Z","timestamp":1643155200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001722","name":"koninklijke nederlandse akademie van wetenschappen","doi-asserted-by":"publisher","award":["PSA SA BD 01"],"award-info":[{"award-number":["PSA SA BD 01"]}],"id":[{"id":"10.13039\/501100001722","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Digit Imaging"],"published-print":{"date-parts":[[2022,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Organs-at-risk contouring is time consuming and labour intensive. Automation by deep learning algorithms would decrease the workload of radiotherapists and technicians considerably. However, the variety of metrics used for the evaluation of deep learning algorithms make the results of many papers difficult to interpret and compare. In this paper, a qualitative evaluation is done on five established metrics to assess whether their values correlate with clinical usability. A total of 377 CT volumes with heart delineations were randomly selected for training and evaluation. A deep learning algorithm was used to predict the contours of the heart. A total of 101 CT slices from the validation set with the predicted contours were shown to three experienced radiologists. They examined each slice independently whether they would accept or adjust the prediction and if there were (small) mistakes. For each slice, the scores of this qualitative evaluation were then compared with the S\u00f8rensen-Dice coefficient (DC), the Hausdorff distance (HD), pixel-wise accuracy, sensitivity and precision. The statistical analysis of the qualitative evaluation and metrics showed a significant correlation. Of the slices with a DC over 0.96 (<jats:italic>N<\/jats:italic>\u2009=\u200920) or a 95% HD under 5 voxels (<jats:italic>N<\/jats:italic>\u2009=\u200925), no slices were rejected by the readers. Contours with lower DC or higher HD were seen in both rejected and accepted contours. Qualitative evaluation shows that it is difficult to use common quantification metrics as indicator for use in clinic. We might need to change the reporting of quantitative metrics to better reflect clinical acceptance.<\/jats:p>","DOI":"10.1007\/s10278-021-00573-9","type":"journal-article","created":{"date-parts":[[2022,1,26]],"date-time":"2022-01-26T20:51:52Z","timestamp":1643230312000},"page":"240-247","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Qualitative Evaluation of Common Quantitative Metrics for Clinical Acceptance of Automatic Segmentation: a Case Study on Heart Contouring from CT Images by Deep Learning Algorithms"],"prefix":"10.1007","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1270-0613","authenticated-orcid":false,"given":"L. B.","family":"van den Oever","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"W. A.","family":"van Veldhuizen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"L. J.","family":"Cornelissen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"D. S.","family":"Spoor","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"T. P.","family":"Willems","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"G.","family":"Kramer","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"T.","family":"Stigter","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"M.","family":"Rook","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"A. P. G.","family":"Crijns","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"M.","family":"Oudkerk","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"R. N. J.","family":"Veldhuis","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"G. H.","family":"de Bock","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"P. M. A.","family":"van Ooijen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,1,26]]},"reference":[{"key":"573_CR1","doi-asserted-by":"publisher","first-page":"25","DOI":"10.3389\/fcvm.2020.00025","volume":"7","author":"C Chen","year":"2020","unstructured":"Chen C, Qin C, Qiu H, et al (2020) Deep Learning for Cardiac Image Segmentation: A Review. Front Cardiovasc Med 7:25. https:\/\/doi.org\/10.3389\/fcvm.2020.00025","journal-title":"Front Cardiovasc Med"},{"key":"573_CR2","doi-asserted-by":"publisher","first-page":"101796","DOI":"10.1016\/j.media.2020.101796","volume":"66","author":"L Maier-Hein","year":"2020","unstructured":"Maier-Hein L, Reinke A, Kozubek M, et al (2020) BIAS: Transparent reporting of biomedical image analysis challenges. Med Image Anal 66:101796. https:\/\/doi.org\/10.1016\/j.media.2020.101796","journal-title":"Med Image Anal"},{"key":"573_CR3","doi-asserted-by":"publisher","first-page":"5217","DOI":"10.1038\/s41467-018-07619-7","volume":"9","author":"L Maier-Hein","year":"2018","unstructured":"Maier-Hein L, Eisenmann M, Reinke A, et al (2018) Why rankings of biomedical image analysis competitions should be interpreted with care. Nat Commun 9:5217. https:\/\/doi.org\/10.1038\/s41467-018-07619-7","journal-title":"Nat Commun"},{"key":"573_CR4","unstructured":"Joyce T, Chartsias A, Tsaftaris SA (2018) Deep Multi-Class Segmentation Without Ground-Truth Labels. In: Medical ImagingWith Deep Learning. pp 1\u20139"},{"key":"573_CR5","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1016\/j.media.2017.11.008","volume":"44","author":"M Zreik","year":"2018","unstructured":"Zreik M, Lessmann N, van Hamersvelt RW, et al (2018) Deep learning analysis of the myocardium in coronary CT angiography for identification of patients with functionally significant coronary artery stenosis. Med Image Anal 44:72\u201385. https:\/\/doi.org\/10.1016\/j.media.2017.11.008","journal-title":"Med Image Anal"},{"key":"573_CR6","doi-asserted-by":"crossref","unstructured":"Wang C, MacGillivray T, Macnaught G, et al (2018) A two-stage 3D Unet framework for multi-class segmentation on full resolution image","DOI":"10.1007\/978-3-030-12029-0_21"},{"key":"573_CR7","doi-asserted-by":"publisher","first-page":"109114","DOI":"10.1016\/j.ejrad.2020.109114","volume":"129","author":"LB van den Oever","year":"2020","unstructured":"van den Oever LB, Cornelissen L, Vonder M, et al (2020) Deep learning for automated exclusion of cardiac CT examinations negative for coronary artery calcium. Eur J Radiol 129:109114. https:\/\/doi.org\/10.1016\/j.ejrad.2020.109114","journal-title":"Eur J Radiol"},{"key":"573_CR8","doi-asserted-by":"publisher","first-page":"5105","DOI":"10.1002\/mp.13200","volume":"45","author":"MJ Gooding","year":"2018","unstructured":"Gooding MJ, Smith AJ, Tariq M, et al (2018) Comparative evaluation of autocontouring in clinical practice: A practical method using the Turing test. Med Phys 45:5105\u20135115. https:\/\/doi.org\/10.1002\/mp.13200","journal-title":"Med Phys"},{"key":"573_CR9","doi-asserted-by":"publisher","DOI":"10.1016\/S2589-7500(19)30123-2","author":"X Liu","year":"2019","unstructured":"Liu X, Faes L, Kale AU, et al (2019) A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Heal. https:\/\/doi.org\/10.1016\/S2589-7500(19)30123-2","journal-title":"Lancet Digit Heal"},{"key":"573_CR10","doi-asserted-by":"publisher","first-page":"1171","DOI":"10.1200\/JCO.2016.69.8480","volume":"35","author":"VAB Van Den Bogaard","year":"2017","unstructured":"Van Den Bogaard VAB, Ta BDP, Van Der Schaaf A, et al (2017) Validation and modification of a prediction model for acute cardiac events in patients with breast cancer treated with radiotherapy based on three-dimensional dose distributions to cardiac substructures. J Clin Oncol 35:1171\u20131178. https:\/\/doi.org\/10.1200\/JCO.2016.69.8480","journal-title":"J Clin Oncol"},{"key":"573_CR11","doi-asserted-by":"publisher","first-page":"312","DOI":"10.1016\/j.radonc.2017.11.012","volume":"126","author":"T Lustberg","year":"2018","unstructured":"Lustberg T, van Soest J, Gooding M, et al (2018) Clinical evaluation of atlas and deep learning based automatic contouring for lung cancer. Radiother Oncol 126:312\u2013317. https:\/\/doi.org\/10.1016\/j.radonc.2017.11.012","journal-title":"Radiother Oncol"},{"key":"573_CR12","doi-asserted-by":"crossref","unstructured":"Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). pp 234\u2013241","DOI":"10.1007\/978-3-319-24574-4_28"}],"container-title":["Journal of Digital Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-021-00573-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10278-021-00573-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-021-00573-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,3,14]],"date-time":"2022-03-14T17:43:44Z","timestamp":1647279824000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10278-021-00573-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,26]]},"references-count":12,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,4]]}},"alternative-id":["573"],"URL":"https:\/\/doi.org\/10.1007\/s10278-021-00573-9","relation":{},"ISSN":["0897-1889","1618-727X"],"issn-type":[{"value":"0897-1889","type":"print"},{"value":"1618-727X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,26]]},"assertion":[{"value":"18 June 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 November 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 December 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 January 2022","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Ethical approval was waived by the local Ethics Committee of the UMCG in view of the retrospective nature of the study and all the procedures being performed were part of the routine care.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics Approval"}},{"value":"All participants signed consent forms for use of their data in research.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to Participate"}},{"value":"All authors and participants agreed with publication of this research.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for Publication"}},{"value":"The authors declare no competing interests.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}}]}}