{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T01:17:38Z","timestamp":1760059058703,"version":"build-2065373602"},"reference-count":49,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2025,5,19]],"date-time":"2025-05-19T00:00:00Z","timestamp":1747612800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Canadian Institutes of Health Research (CIHR)","award":["437221"],"award-info":[{"award-number":["437221"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Real-time tumor-tracked radiotherapy with a linear accelerator-magnetic resonance (linac-MR) hybrid system requires accurate tumor delineation at a fast MR imaging rate. Various autocontouring methods have been previously evaluated against \u201cgold standard\u201d manual contours by experts. However, manually drawn contours have inherent intra- and inter-observer variations. We aim to quantify these variations and evaluate our tumor-autocontouring algorithm against the manual contours. Ten liver, ten prostate, and ten lung cancer patients were scanned using a 3 tesla (T) magnetic resonance imaging (MRI) scanner with a 2D balanced steady-state free precession (bSSFP) sequence at 4 frames\/s. Three experts manually contoured the tumor in two sessions. For autocontouring, an in-house built U-Net-based autocontouring algorithm was used, whose hyperparameters were optimized for each patient, expert, and session (PES). For evaluation, (A) Automatic vs. Manual and (B) Manual vs. Manual contour comparisons were performed. For (A) and (B), three types of comparisons were performed: (a) same expert same session, (b) same expert different session, and (c) different experts, using Dice coefficient (DC), centroid displacement (CD), and the Hausdorff distance (HD). For (A), the algorithm was trained using one expert\u2019s contours and its autocontours were compared to contours from (a)\u2013(c). For Automatic vs. Manual evaluations (Aa\u2013Ac), DC = 0.91, 0.86, 0.78, CD = 1.3, 1.8, 2.7 mm, and HD = 3.1, 4.6, 7.0 mm averaged over 30 patients were achieved, respectively. For Manual vs. Manual evaluations (Ba\u2013Bc), DC = 1.00, 0.85, 0.77, CD = 0.0, 2.1, 2.8 mm, and HD = 0.0, 4.9, 7.2 mm were achieved, respectively. We have quantified the intra- and inter-observer variations in manual contouring of liver, prostate, and lung patients. Our PES-specific optimized algorithm generated autocontours with agreement levels comparable to these manual variations, but with high efficiency (54 ms\/autocontour vs. 9 s\/manual contour).<\/jats:p>","DOI":"10.3390\/a18050290","type":"journal-article","created":{"date-parts":[[2025,5,19]],"date-time":"2025-05-19T05:37:13Z","timestamp":1747633033000},"page":"290","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Quantifying Intra- and Inter-Observer Variabilities in Manual Contours for Radiotherapy: Evaluation of an MR Tumor Autocontouring Algorithm for Liver, Prostate, and Lung Cancer Patients"],"prefix":"10.3390","volume":"18","author":[{"given":"Gawon","family":"Han","sequence":"first","affiliation":[{"name":"Medical Physics Division, Department of Oncology, University of Alberta, 11560 University Avenue, Edmonton, AB T6G 1Z2, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Arun","family":"Elangovan","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Cross Cancer Institute, 11560 University Avenue, Edmonton, AB T6G 1Z2, Canada"},{"name":"Department of Oncology, Radiation Oncology, Tom Baker Cancer Centre, University of Calgary, 1331 29th Street, NW Calgary, AB T2N 4N2, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5644-3626","authenticated-orcid":false,"given":"Jordan","family":"Wong","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Cross Cancer Institute, 11560 University Avenue, Edmonton, AB T6G 1Z2, Canada"},{"name":"Radiation Oncology, British Columbia Cancer\u2014Victoria, 2410 Lee Avenue, Victoria, BC V8R 6V5, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Asmara","family":"Waheed","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Cross Cancer Institute, 11560 University Avenue, Edmonton, AB T6G 1Z2, Canada"},{"name":"Radiation Oncology Division, Department of Oncology, University of Alberta, 11560 University Avenue, Edmonton, AB T6G 1Z2, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1330-5562","authenticated-orcid":false,"given":"Keith","family":"Wachowicz","sequence":"additional","affiliation":[{"name":"Medical Physics Division, Department of Oncology, University of Alberta, 11560 University Avenue, Edmonton, AB T6G 1Z2, Canada"},{"name":"Department of Medical Physics, Cross Cancer Institute, 11560 University Avenue, Edmonton, AB T6G 1Z2, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nawaid","family":"Usmani","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Cross Cancer Institute, 11560 University Avenue, Edmonton, AB T6G 1Z2, Canada"},{"name":"Radiation Oncology Division, Department of Oncology, University of Alberta, 11560 University Avenue, Edmonton, AB T6G 1Z2, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7198-1876","authenticated-orcid":false,"given":"Zsolt","family":"Gabos","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Cross Cancer Institute, 11560 University Avenue, Edmonton, AB T6G 1Z2, Canada"},{"name":"Radiation Oncology Division, Department of Oncology, University of Alberta, 11560 University Avenue, Edmonton, AB T6G 1Z2, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6113-0528","authenticated-orcid":false,"given":"Jihyun","family":"Yun","sequence":"additional","affiliation":[{"name":"Medical Physics Division, Department of Oncology, University of Alberta, 11560 University Avenue, Edmonton, AB T6G 1Z2, Canada"},{"name":"Department of Medical Physics, Cross Cancer Institute, 11560 University Avenue, Edmonton, AB T6G 1Z2, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4402-937X","authenticated-orcid":false,"given":"B. Gino","family":"Fallone","sequence":"additional","affiliation":[{"name":"Medical Physics Division, Department of Oncology, University of Alberta, 11560 University Avenue, Edmonton, AB T6G 1Z2, Canada"},{"name":"Department of Medical Physics, Cross Cancer Institute, 11560 University Avenue, Edmonton, AB T6G 1Z2, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2025,5,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"349","DOI":"10.1016\/j.radonc.2004.07.017","article-title":"Measurement of tumor diameter-dependent mobility of lung tumors by dynamic MRI","volume":"73","author":"Plathow","year":"2004","journal-title":"Radiother. Oncol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1229","DOI":"10.1016\/j.ijrobp.2005.11.016","article-title":"Speed and amplitude of lung tumor motion precisely detected in fourdimensional setup and in real-time tumor-tracking radiotherapy","volume":"64","author":"Shirato","year":"2006","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1053\/j.semradonc.2003.10.008","article-title":"Intrafractional tumor motion: Lung and liver","volume":"14","author":"Shirato","year":"2004","journal-title":"Semin. Radiat. Oncol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"836","DOI":"10.1259\/bjr\/29483804","article-title":"Quantitative analysis of lung and tumour mobility: Comparison of two time-resolved MRI sequences","volume":"78","author":"Plathow","year":"2005","journal-title":"Br. J. Radiol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2084","DOI":"10.1118\/1.3125662","article-title":"First MR images obtained during megavoltage photon irradiation from a prototype integrated linac-MR system","volume":"36","author":"Fallone","year":"2009","journal-title":"Med. Phys."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1016\/j.semradonc.2014.02.011","article-title":"The rotating biplanarlinac magnetic resonance imaging system","volume":"24","author":"Fallone","year":"2014","journal-title":"Semin. Radiat. Oncol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"196","DOI":"10.1016\/j.semradonc.2014.02.008","article-title":"The ViewRay system: Magnetic resonance-guided and controlled radiotherapy","volume":"24","author":"Mutic","year":"2014","journal-title":"Semin. Radiat. Oncol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"N229","DOI":"10.1088\/0031-9155\/54\/12\/N01","article-title":"Integrating a 1.5 T MRI scanner with a 6 MV accelerator: Proof of concept","volume":"54","author":"Raaymakers","year":"2009","journal-title":"Phys. Med. Biol."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"051718","DOI":"10.1118\/1.4802735","article-title":"First demonstration of intrafractional tumor-tracked irradiation using 2D phantom MR images on a prototype linac-MR","volume":"40","author":"Yun","year":"2013","journal-title":"Med. Phys."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"753","DOI":"10.1118\/1.3284543","article-title":"Real-time tumor tracking: Automatic compensation of target motion using the Siemens 160 MLC","volume":"37","author":"Tacke","year":"2010","journal-title":"Med. Phys."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"859","DOI":"10.1016\/j.ijrobp.2009.02.012","article-title":"First demonstration of combined kV\/MV image-guided real-time dynamic multileaf-collimator target tracking","volume":"74","author":"Cho","year":"2009","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2050","DOI":"10.1118\/1.2905355","article-title":"Management of three-dimensional intrafraction motion through real-time DMLC tracking","volume":"35","author":"Sawant","year":"2008","journal-title":"Med. Phys."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"3874","DOI":"10.1118\/1.2349696","article-title":"The management of respiratory motion in radiation oncology report of AAPM Task Group 76","volume":"33","author":"Keall","year":"2006","journal-title":"Med. Phys."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"4631","DOI":"10.1002\/mp.13748","article-title":"Technical note: Realtime 3D MRI in the presence of motion for MRI-guided radiotherapy: 3D dynamic keyhole imaging with super-resolution","volume":"46","author":"Kim","year":"2019","journal-title":"Med. Phys."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"4943","DOI":"10.1088\/0031-9155\/58\/14\/4943","article-title":"Three-dimensional MRI-linac intra-fraction guidance using multiple orthogonal cine-MRI planes","volume":"58","author":"Bjerre","year":"2013","journal-title":"Phys. Med. Biol."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"2296","DOI":"10.1118\/1.4916657","article-title":"Neural-network based autocontouring algorithm for intrafractional lung-tumor tracking using Linac-MR","volume":"42","author":"Yun","year":"2015","journal-title":"Med. Phys."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"5161","DOI":"10.1118\/1.4961403","article-title":"A particle filter based autocontouring algorithm for lung tumor tracking using dynamic magnetic resonance imaging","volume":"43","author":"Bourque","year":"2016","journal-title":"Med. Phys."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"587","DOI":"10.1002\/mp.14659","article-title":"Stability of conventional and machine learning-based tumor auto-segmentation techniques using undersampled dynamic radial bSSFP acquisitions on a 0.35 T hybrid MR-linac system","volume":"48","author":"Friedrich","year":"2021","journal-title":"Med. Phys."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"485","DOI":"10.1016\/j.radonc.2017.09.013","article-title":"Tumor auto-contouring on 2d cine MRI for locally advanced lung cancer: A comparative study","volume":"125","author":"Fast","year":"2017","journal-title":"Radiother. Oncol."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Menten, M.J., Fast, M.F., Wetscherek, A., Rank, C.M., Kachelrie\u00df, M., Collins, D.J., Nill, S., and Oelfke, U. (2018). The impact of 2D cine MR imaging parameters on automated tumor and organ localization for MR-guided real-time adaptive radiotherapy. Phys. Med. Biol., 63.","DOI":"10.1088\/1361-6560\/aae74d"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"3773","DOI":"10.1088\/0031-9155\/56\/13\/003","article-title":"MRI-guided tumor tracking in lung cancer radiotherapy","volume":"56","author":"Du","year":"2011","journal-title":"Phys. Med. Biol."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"712","DOI":"10.1109\/42.796284","article-title":"Nonrigid registration using free-form deformations: Application to breast MR images","volume":"18","author":"Rueckert","year":"1999","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1016\/S1361-8415(98)80022-4","article-title":"Image matching as a diffusion process: An analogy with Maxwell\u2019s demons","volume":"2","author":"Thirion","year":"1998","journal-title":"Med. Image Anal."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"S61","DOI":"10.1016\/j.neuroimage.2008.10.040","article-title":"Diffeomorphic demons: Efficient non-parametric image registration","volume":"45","author":"Vercauteren","year":"2009","journal-title":"Neuroimage"},{"key":"ref_25","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5\u20139). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention\u2013MICCAI 2015: 18th International Conference, Munich, Germany. Proceedings, Part III 18."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1016\/j.radonc.2019.10.019","article-title":"Comparing deep learning-based auto-segmentation of organs at risk and clinical target volumes to expert inter-observer variability in radiotherapy planning","volume":"144","author":"Wong","year":"2020","journal-title":"Radiother. Oncol."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1186\/s13014-021-01896-1","article-title":"Evaluation of auto-segmentation accuracy of cloud-based artificial intelligence and atlas-based models","volume":"16","author":"Urago","year":"2021","journal-title":"Radiat. Oncol."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1016\/j.diii.2023.08.001","article-title":"Reference standard for the evaluation of automatic segmentation algorithms: Quantification of inter observer variability of manual delineation of prostate contour on MRI","volume":"105","author":"Hamzaoui","year":"2024","journal-title":"Diagn. Interv. Imaging"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1826","DOI":"10.1007\/s10278-023-00800-5","article-title":"Lossy Image Compression in a Preclinical Multimodal Imaging Study","volume":"36","author":"Cunha","year":"2023","journal-title":"J. Digit. Imaging"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"97","DOI":"10.5603\/RPOR.a2022.0004","article-title":"Evaluation of inter- and intra-observer variations in prostate gland delineation using CT-alone versus CT\/TPUS","volume":"27","author":"Lim","year":"2022","journal-title":"Rep. Pract. Oncol. Radiother."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"100437","DOI":"10.1016\/j.phro.2023.100437","article-title":"Accuracy of deformable image registration-based intra-fraction motion management in Magnetic Resonance-guided radiotherapy","volume":"26","author":"Palacios","year":"2023","journal-title":"Phys. Imaging Radiat. Oncol."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1002\/mp.12687","article-title":"Evaluating performance of a user-trained MR lung tumor autocontouring algorithm in the context of intra- and interobserver variations","volume":"45","author":"Yip","year":"2018","journal-title":"Med. Phys."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"602","DOI":"10.1016\/j.ijrobp.2010.04.028","article-title":"Comparison of liver tumor motion with and without abdominal compression using cine-magnetic resonance imaging","volume":"79","author":"Eccles","year":"2011","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"5013","DOI":"10.1120\/jacmp.v16i2.5013","article-title":"Intrafractional prostate motion during external beam radiotherapy monitored by a real-time target localization system","volume":"16","author":"Tong","year":"2015","journal-title":"J. Appl. Clin. Med. Phys."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1323","DOI":"10.1016\/j.mri.2012.05.001","article-title":"3D Slicer as an image computing platform for the quantitative imaging network","volume":"30","author":"Fedorov","year":"2012","journal-title":"Magn. Reson. Imaging"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Han, G., Wachowicz, K., Usmani, N., Yee, D., Wong, J., Elangovan, A., Yun, J., and Fallone, B.G. (2025). Patient-specific hyperparameter optimization of a deep learning-based tumor autocontouring algorithm on 2D liver, prostate, and lung cine MR images: A pilot study. Algorithms, 18.","DOI":"10.3390\/a18040233"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1162\/106365601750190398","article-title":"Completely Derandomized Self-Adaptation in Evolution Strategies","volume":"9","author":"Hansen","year":"2001","journal-title":"Evol. Comput."},{"key":"ref_38","first-page":"e576","article-title":"An AI-based tumor autocontouring algorithm for non-invasive intra-fractional tumor-tracked radiotherapy (nifteRT) on linac-MR","volume":"47","author":"Yun","year":"2020","journal-title":"Med. Phys."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1038\/s41592-020-01008-z","article-title":"nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation","volume":"18","author":"Isensee","year":"2021","journal-title":"Nat. Methods"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Taha, A.A., and Hanbury, A. (2015). Metrics for evaluating 3D medical image segmentation: Analysis, selection, and tool. BMC Med. Imaging, 15.","DOI":"10.1186\/s12880-015-0068-x"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"297","DOI":"10.2307\/1932409","article-title":"Measures of the amount of ecologic association between species","volume":"26","author":"Dice","year":"1945","journal-title":"Ecology"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"850","DOI":"10.1109\/34.232073","article-title":"Comparing images using the Hausdorff distance","volume":"15","author":"Huttenlocher","year":"1993","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1016\/j.radonc.2009.12.028","article-title":"Inter-observer and intraobserver reliability for lung cancer target volume delineation in the 4DCT era","volume":"95","author":"Louie","year":"2010","journal-title":"Radiother. Oncol."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1016\/S0936-6555(05)81075-1","article-title":"Treatment and planning decisions in non-small cell carcinoma of the lung: An Australasian patterns of practice study","volume":"4","author":"Hamilton","year":"1992","journal-title":"Clin. Oncol. (R. Coll. Radiol.)"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1016\/S0167-8140(99)00143-7","article-title":"Evaluation of a target contouring protocol for 3D conformal radiotherapy in non-small cell lung cancer","volume":"53","author":"Senan","year":"1999","journal-title":"Radiother. Oncol."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1186\/s40658-022-00515-6","article-title":"Intra- and inter-operator variability in MRI-based manual segmentation of HCC lesions and its impact on dosimetry","volume":"9","author":"Covert","year":"2022","journal-title":"EJNMMI Phys."},{"key":"ref_47","first-page":"98","article-title":"Technical design and concept of a 0.35 T MR-Linac","volume":"18","year":"2019","journal-title":"Clin. Transl. Radiat. Oncol."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"4148","DOI":"10.1002\/mp.13717","article-title":"MRI super-resolution reconstruction for MRI-guided adaptive radiotherapy using cascaded deep learning: In the presence of limited training data and unknown translation model","volume":"46","author":"Chun","year":"2019","journal-title":"Med. Phys."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1038\/s43856-024-00489-9","article-title":"Super-resolution neural networks improve the spatiotemporal resolution of adaptive MRI-guided radiation therapy","volume":"4","author":"Grover","year":"2024","journal-title":"Commun. Med."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/18\/5\/290\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:35:00Z","timestamp":1760031300000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/18\/5\/290"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,19]]},"references-count":49,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2025,5]]}},"alternative-id":["a18050290"],"URL":"https:\/\/doi.org\/10.3390\/a18050290","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2025,5,19]]}}}