{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T15:49:24Z","timestamp":1753890564226,"version":"3.41.2"},"reference-count":31,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2022,2,15]],"date-time":"2022-02-15T00:00:00Z","timestamp":1644883200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100006041","name":"Innovate UK","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100006041","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Cardiovasc. Med."],"abstract":"<jats:sec><jats:title>Background<\/jats:title><jats:p>The quantitative measures used to assess the performance of automated methods often do not reflect the clinical acceptability of contouring. A quality-based assessment of automated cardiac magnetic resonance (CMR) segmentation more relevant to clinical practice is therefore needed.<\/jats:p><\/jats:sec><jats:sec><jats:title>Objective<\/jats:title><jats:p>We propose a new method for assessing the quality of machine learning (ML) outputs. We evaluate the clinical utility of the proposed method as it is employed to systematically analyse the quality of an automated contouring algorithm.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>A dataset of short-axis (SAX) cine CMR images from a clinically heterogeneous population (<jats:italic>n<\/jats:italic> = 217) were manually contoured by a team of experienced investigators. On the same images we derived automated contours using a ML algorithm. A contour quality scoring application randomly presented manual and automated contours to four blinded clinicians, who were asked to assign a quality score from a predefined rubric. Firstly, we analyzed the distribution of quality scores between the two contouring methods across all clinicians. Secondly, we analyzed the interobserver reliability between the raters. Finally, we examined whether there was a variation in scores based on the type of contour, SAX slice level, and underlying disease.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The overall distribution of scores between the two methods was significantly different, with automated contours scoring better than the manual (OR (95% CI) = 1.17 (1.07\u20131.28), <jats:italic>p<\/jats:italic> = 0.001; <jats:italic>n<\/jats:italic> = 9401). There was substantial scoring agreement between raters for each contouring method independently, albeit it was significantly better for automated segmentation (automated: AC2 = 0.940, 95% CI, 0.937\u20130.943 vs manual: AC2 = 0.934, 95% CI, 0.931\u20130.937; <jats:italic>p<\/jats:italic> = 0.006). Next, the analysis of quality scores based on different factors was performed. Our approach helped identify trends patterns of lower segmentation quality as observed for left ventricle epicardial and basal contours with both methods. Similarly, significant differences in quality between the two methods were also found in dilated cardiomyopathy and hypertension.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>Our results confirm the ability of our systematic scoring analysis to determine the clinical acceptability of automated contours. This approach focused on the contours' clinical utility could ultimately improve clinicians' confidence in artificial intelligence and its acceptability in the clinical workflow.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fcvm.2021.816985","type":"journal-article","created":{"date-parts":[[2022,2,15]],"date-time":"2022-02-15T07:20:56Z","timestamp":1644909656000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["A Systematic Quality Scoring Analysis to Assess Automated Cardiovascular Magnetic Resonance Segmentation Algorithms"],"prefix":"10.3389","volume":"8","author":[{"given":"Elisa","family":"Rauseo","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Muhammad","family":"Omer","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alborz","family":"Amir-Khalili","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alireza","family":"Sojoudi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Thu-Thao","family":"Le","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stuart Alexander","family":"Cook","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Derek John","family":"Hausenloy","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Briana","family":"Ang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Desiree-Faye","family":"Toh","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jennifer","family":"Bryant","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Calvin Woon Loong","family":"Chin","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jose Miguel","family":"Paiva","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kenneth","family":"Fung","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jackie","family":"Cooper","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mohammed Yunus","family":"Khanji","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nay","family":"Aung","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Steffen Erhard","family":"Petersen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2022,2,15]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"174","DOI":"10.1136\/heartjnl-2019-314856","article-title":"Cardiovascular magnetic resonance: applications and practical considerations for the general cardiologist","volume":"106","author":"Arnold","year":"2019","journal-title":"Heart."},{"key":"B2","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1007\/s00117-020-00687-1","article-title":"Cardiac MRI\u2014Update 2020","volume":"60","author":"Busse","year":"2020","journal-title":"Radiologe."},{"key":"B3","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1007\/s11897-008-0021-1","article-title":"Application of cardiac magnetic resonance imaging in cardiomyopathy","volume":"5","author":"Steel","year":"2008","journal-title":"Curr Heart Fail Rep."},{"key":"B4","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1007\/s10554-017-1116-0","article-title":"Role of cardiovascular magnetic resonance in acute and chronic ischemic heart disease","volume":"34","author":"Baritussio","year":"2018","journal-title":"Int J Cardiovasc Imaging."},{"key":"B5","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1016\/j.hfc.2020.08.001","article-title":"Cardiac magnetic resonance quantification of structure-function relationships in heart failure","volume":"17","author":"Nguyen","year":"2021","journal-title":"Heart Fail Clin."},{"key":"B6","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1007\/s00276-008-0415-5","article-title":"Assessment of left ventricular volumes and function by cine-MR imaging depending on the investigator's experience","volume":"31","author":"Bailly","year":"2009","journal-title":"Surg Radiol Anat."},{"key":"B7","doi-asserted-by":"publisher","first-page":"2514","DOI":"10.1109\/TMI.2018.2837502","article-title":"Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved?","volume":"37","author":"Bernard","year":"2018","journal-title":"IEEE Trans Med Imaging"},{"key":"B8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12968-019-0575-y","article-title":"Machine learning in cardiovascular magnetic resonance: Basic concepts and applications","volume":"21","author":"Leiner","year":"2019","journal-title":"J Cardiovasc Magn Reson."},{"key":"B9","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1186\/s12968-018-0471-x","article-title":"Automated cardiovascular magnetic resonance image analysis with fully convolutional networks 08 Information and Computing Sciences 0801 Artificial Intelligence and Image Processing","volume":"20","author":"Bai","year":"2018","journal-title":"J Cardiovasc Magn Reson"},{"key":"B10","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1016\/j.media.2019.05.006","article-title":"Quantitative CMR population imaging on 20,000 subjects of the UK biobank imaging study: LV\/RV quantification pipeline and its evaluation","volume":"56","author":"Attar","year":"2019","journal-title":"Med Image Anal."},{"key":"B11","doi-asserted-by":"publisher","first-page":"504","DOI":"10.1016\/j.jacr.2017.12.026","article-title":"Artificial Intelligence and machine learning in radiology: opportunities, challenges, pitfalls, and criteria for success","volume":"15","author":"Thrall","year":"2018","journal-title":"J Am Coll Radiol."},{"key":"B12","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1186\/s12880-015-0068-x","article-title":"Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool","volume":"15","author":"Taha","year":"2015","journal-title":"BMC Med Imaging"},{"key":"B13","doi-asserted-by":"publisher","first-page":"1451","DOI":"10.1109\/TMI.2006.880587","article-title":"Generalized overlap measures for evaluation and validation in medical image analysis","volume":"25","author":"Crum","year":"2006","journal-title":"IEEE Trans Med Imaging."},{"key":"B14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12968-016-0236-3","article-title":"Cardiovascular magnetic resonance reference ranges for the heart and aorta in Chinese at 3T","volume":"18","author":"Le","year":"2016","journal-title":"J Cardiovasc Magn Reson"},{"key":"B15","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-24574-4_28","article-title":"U-Net: convolutional networks for biomedical image segmentation","volume-title":"Medical Image Computing and Computer-Assisted Intervention \u2013 MICCAI 2015. Lecture Notes in Computer Science","author":"Ronneberger","year":"2015"},{"key":"B16","doi-asserted-by":"publisher","first-page":"1001779","DOI":"10.1371\/journal.pmed.1001779","article-title":"UK Biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age","volume":"12","author":"Sudlow","year":"2015","journal-title":"PLoS Med."},{"key":"B17","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1186\/1532-429X-15-46","article-title":"Imaging in population science: cardiovascular magnetic resonance in 100,000 participants of UK Biobank - rationale, challenges and approaches","volume":"15","author":"Petersen","year":"2013","journal-title":"J Cardiovasc Magn Reson."},{"key":"B18","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1186\/s12968-017-0327-9","article-title":"Reference ranges for cardiac structure and function using cardiovascular magnetic resonance (CMR) in Caucasians from the UK Biobank population cohort","volume":"19","author":"Petersen","year":"2017","journal-title":"J Cardiovasc Magn Reson."},{"volume-title":"Handbook of Inter-rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters","year":"2014","author":"Gwet","key":"B19"},{"key":"B20","unstructured":"Disagreement on agreement: two alternative agreement coefficients112\n            BloodE\n            SprattKF\n          18663485SAS Global Forum, Orlando, Florida1062007"},{"key":"B21","doi-asserted-by":"publisher","first-page":"363","DOI":"10.2307\/2529786","article-title":"An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics."},{"key":"B22","doi-asserted-by":"publisher","first-page":"178","DOI":"10.1016\/S1076-6332(03)00671-8","article-title":"Statistical validation of image segmentation quality based on a spatial overlap index","volume":"11","author":"Zou","year":"2004","journal-title":"Acad Radiol."},{"key":"B23","doi-asserted-by":"publisher","first-page":"7186","DOI":"10.1109\/IEMBS.2005.1616166","article-title":"Evaluation of segmentation algorithms for medical imaging","volume":"7","author":"Fenster","year":"2005","journal-title":"Annu Int Conf IEEE Eng Med Biol - Proc."},{"key":"B24","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1145\/1150402.1150422","article-title":"Reverse testing: an efficient framework to select amongst classifiers under sample selection bias","volume-title":"Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '06","author":"Fan","year":"2006"},{"key":"B25","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-15939-8_35","article-title":"Cross validation framework to choose amongst models and datasets for transfer learning","volume-title":"Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2010. Lecture Notes in Computer Science","author":"Zhong","year":"2010"},{"key":"B26","doi-asserted-by":"publisher","first-page":"528","DOI":"10.1007\/978-3-642-33415-3_65","article-title":"Evaluating segmentation error without ground truth","volume":"15","author":"Kohlberger","year":"2012","journal-title":"Med Image Comput Comput Assist Interv"},{"key":"B27","first-page":"49","article-title":"Estimating prediction qualities without ground truth: a revisit of the reverse testing framework","volume-title":"IEEE 24th International Conference on Pattern Recognition (ICPR)","author":"Bhaskaruni","year":"2018"},{"key":"B28","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1186\/s12968-019-0523-x","article-title":"Automated quality control in image segmentation: application to the UK Biobank cardiovascular magnetic resonance imaging study","volume":"21","author":"Robinson","year":"2019","journal-title":"J Cardiovasc Magn Reson."},{"key":"B29","doi-asserted-by":"publisher","first-page":"5105","DOI":"10.1002\/mp.13200","article-title":"Comparative evaluation of autocontouring in clinical practice: a practical method using the Turing test","volume":"45","author":"Gooding","year":"2018","journal-title":"Med Phys."},{"key":"B30","doi-asserted-by":"publisher","first-page":"433","DOI":"10.1093\/mind\/LIX.236.433","article-title":"Computing machinery and intelligence","volume":"59","author":"Turing","year":"1950","journal-title":"MIND"},{"key":"B31","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1145\/141420.141422","article-title":"The turing test is not a trick: turing indistinguishability is a scientific criterion","volume":"3","author":"Harnad","year":"1992","journal-title":"ACM SIGART Bull."}],"container-title":["Frontiers in Cardiovascular Medicine"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fcvm.2021.816985\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,2,15]],"date-time":"2022-02-15T07:21:05Z","timestamp":1644909665000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fcvm.2021.816985\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,15]]},"references-count":31,"alternative-id":["10.3389\/fcvm.2021.816985"],"URL":"https:\/\/doi.org\/10.3389\/fcvm.2021.816985","relation":{},"ISSN":["2297-055X"],"issn-type":[{"type":"electronic","value":"2297-055X"}],"subject":[],"published":{"date-parts":[[2022,2,15]]},"article-number":"816985"}}