{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,13]],"date-time":"2025-12-13T14:31:33Z","timestamp":1765636293609,"version":"3.48.0"},"reference-count":25,"publisher":"IOP Publishing","issue":"5","license":[{"start":{"date-parts":[[2020,3,6]],"date-time":"2020-03-06T00:00:00Z","timestamp":1583452800000},"content-version":"vor","delay-in-days":5,"URL":"https:\/\/publishingsupport.iopscience.iop.org\/iop-standard\/v1"},{"start":{"date-parts":[[2020,3,6]],"date-time":"2020-03-06T00:00:00Z","timestamp":1583452800000},"content-version":"tdm","delay-in-days":5,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"crossref","award":["Canada Graduate Scholarship"],"award-info":[{"award-number":["Canada Graduate Scholarship"]}],"id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Phys. Med. Biol."],"published-print":{"date-parts":[[2020,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Algorithm benchmarking and characterization are an important part of algorithm development and validation prior to clinical implementation. However, benchmarking may be limited to a small collection of test cases due to the resource-intensive nature of establishing \u2018ground-truth\u2019 references. This study proposes a framework for selecting test cases to assess algorithm and workflow equivalence. Effective test case selection may minimize the number of ground-truth comparisons required to establish robust and clinically relevant benchmarking and characterization results.<\/jats:p>\n                  <jats:p>To demonstrate the proposed framework, we clustered differences between two independent workflows estimating during-treatment dose objective violations for 15 head and neck cancer patients (15 planning CTs, 105 on-unit CBCTs). Each workflow used a different deformable image registration algorithm to estimate inter-fractional anatomy and contour changes. The Hopkins statistic tested whether workflow output was inherently clustered and k-medoid clustering formalized cluster assignment. Further statistical analyses verified the relevance of clusters to algorithm output. Data at cluster centers (\u2018medoids\u2019) were considered as candidate test cases representative of workflow-relevant algorithm differences.<\/jats:p>\n                  <jats:p>\n                    The framework indicated that differences in estimated dose objective violations were naturally grouped (Hopkins\u2009\u2009=\u2009\u20090.75, providing 90% confidence). K-medoid clustering identified five clusters which stratified workflow differences (MANOVA:\n                    <jats:italic>p<\/jats:italic>\n                    \u200a\u2009\u2009&lt;\u2009\u20090.001) in estimated parotid gland D50%, spinal cord\/brainstem Dmax, and high dose CTV coverage dose violations (Kendall\u2019s tau:\n                    <jats:italic>p<\/jats:italic>\n                    \u200a\u2009\u2009&lt;\u2009\u20090.05). Systematic algorithm differences resulting in workflow discrepancies were: parotid gland volumes (ANOVA:\n                    <jats:italic>p<\/jats:italic>\n                    \u200a\u2009\u2009&lt;\u2009\u20090.001), external contour deformations (\n                    <jats:italic>t<\/jats:italic>\n                    -test:\n                    <jats:italic>p<\/jats:italic>\n                    \u200a\u2009\u2009=\u2009\u20090.022), and CTV-to-PTV margins (\n                    <jats:italic>t<\/jats:italic>\n                    -test: 0.009), respectively. Five candidate test cases were verified as representative of the five clusters.\n                  <\/jats:p>\n                  <jats:p>The framework successfully clustered workflow outputs and identified five test cases representative of clinically relevant algorithm discrepancies. This approach may improve the allocation of resources during the benchmarking and characterization process and the applicability of results to clinical data.<\/jats:p>","DOI":"10.1088\/1361-6560\/ab6e54","type":"journal-article","created":{"date-parts":[[2020,1,21]],"date-time":"2020-01-21T17:45:24Z","timestamp":1579628724000},"page":"055014","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Data clustering to select clinically-relevant test cases for algorithm benchmarking and characterization"],"prefix":"10.1088","volume":"65","author":[{"given":"Sarah","family":"Weppler","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Colleen","family":"Schinkel","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Charles","family":"Kirkby","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wendy","family":"Smith","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"266","published-online":{"date-parts":[[2020,3,6]]},"reference":[{"key":"pmbab6e54bib001","doi-asserted-by":"publisher","first-page":"289","DOI":"10.1111\/j.2517-6161.1995.tb02031.x","type":"journal-article","article-title":"Controlling the false discovery rate: a practical and powerful approach to multiple testing","volume":"57","author":"Benjamini","year":"1995","journal-title":"J. R. Stat. Soc. B"},{"key":"pmbab6e54bib002","doi-asserted-by":"publisher","first-page":"e43","DOI":"10.1002\/mp.12256","type":"journal-article","article-title":"Use of image registration and fusion algorithms and techniques in radiotherapy: report of the AAPM Radiation Therapy Committee Task Group No. 132","volume":"44","author":"Brock","year":"2017","journal-title":"Med. Phys."},{"key":"pmbab6e54bib003","doi-asserted-by":"publisher","first-page":"3282","DOI":"10.1118\/1.4924176","type":"journal-article","article-title":"SU-E-J-89: comparative analysis of MIM and velocity\u2019s image deformation algorithm using simulated kV-CBCT images for quality assurance","volume":"42","author":"Cline","year":"2015","journal-title":"Med. Phys."},{"key":"pmbab6e54bib004","doi-asserted-by":"publisher","first-page":"5850","DOI":"10.1118\/1.3496356","type":"journal-article","article-title":"Use of a realistic breathing lung phantom to evaluate dose delivery errors","volume":"37","author":"Court","year":"2010","journal-title":"Med. Phys."},{"year":"1973","author":"Duda","key":"pmbab6e54bib005","type":"book"},{"key":"pmbab6e54bib006","doi-asserted-by":"publisher","first-page":"5763","DOI":"10.1088\/0031-9155\/59\/19\/5763","type":"journal-article","article-title":"Patient specific 3D printed phantom for IMRT quality assurance","volume":"59","author":"Ehler","year":"2014","journal-title":"Phys. Med. Biol."},{"key":"pmbab6e54bib007","doi-asserted-by":"publisher","first-page":"4445","DOI":"10.1088\/0031-9155\/55\/16\/S02","type":"journal-article","article-title":"Dosimetric verification and clinical evaluation of a new commercially available Monte Carlo-based dose algorithm for application in stereotactic body radiation therapy (SBRT) treatment planning","volume":"55","author":"Fragoso","year":"2010","journal-title":"Phys. Med. Biol."},{"year":"2012","author":"Han","key":"pmbab6e54bib008","type":"book"},{"key":"pmbab6e54bib009","doi-asserted-by":"publisher","first-page":"769","DOI":"10.1007\/s11704-016-6112-3","type":"journal-article","article-title":"Test-case prioritization: achievements and challenges","volume":"10","author":"Hao","year":"2016","journal-title":"Frontiers Comput. Sci."},{"key":"pmbab6e54bib010","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-84858-7","type":"book","author":"Hastie","year":"2009","edition":"2nd edn"},{"key":"pmbab6e54bib011","doi-asserted-by":"publisher","first-page":"309","DOI":"10.1111\/j.1467-9876.2012.01066.x","type":"journal-article","article-title":"How to find an appropriate clustering for mixed-type variables with application to socio-economic stratification","volume":"62","author":"Hennig","year":"2013","journal-title":"J. R. Stat. Soc. C"},{"key":"pmbab6e54bib012","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1120\/jacmp.v15i1.4564","type":"journal-article","article-title":"Accuracy quantification of a deformable image registration tool applied in a clinical setting","volume":"15","author":"Hoffmann","year":"2014","journal-title":"J. Appl. Clin. Med. Phys."},{"key":"pmbab6e54bib013","doi-asserted-by":"publisher","first-page":"5944","DOI":"10.1118\/1.3013563","type":"journal-article","article-title":"Objective assessment of deformable image registration in radiotherapy: a multi-institutional study","volume":"35","author":"Kashani","year":"2008","journal-title":"Med. Phys."},{"year":"1990","author":"Kaufman","key":"pmbab6e54bib014","doi-asserted-by":"publisher","DOI":"10.1002\/9780470316801","type":"book"},{"key":"pmbab6e54bib015","doi-asserted-by":"publisher","DOI":"10.1118\/1.4901409","type":"journal-article","article-title":"Deformable image registration based automatic CT-to-CT contour propagation for head and neck adaptive radiotherapy in the routine clinical setting","volume":"41","author":"Kumarasiri","year":"2014","journal-title":"Med. Phys."},{"key":"pmbab6e54bib016","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1120\/jacmp.v8i4.2432","type":"journal-article","article-title":"Quantitative evaluation of a cone-beam computed tomography-planning computed tomography deformable image registration method for adaptive radiation therapy","volume":"8","author":"Lawson","year":"2007","journal-title":"J. Appl. Clin. Med. Phys."},{"key":"pmbab6e54bib017","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1021\/ci00065a010","type":"journal-article","article-title":"New index for clustering tendency and its application to chemical problems","volume":"30","author":"Lawson","year":"1990","journal-title":"J. Chem. Inf. Comput. Sci."},{"key":"pmbab6e54bib018","doi-asserted-by":"publisher","first-page":"748","DOI":"10.1002\/mp.12737","type":"journal-article","article-title":"Performance of commercially available deformable image registration platforms for contour propagation using patient-based computational phantoms: a multi-institutional study","volume":"45","author":"Loi","year":"2018","journal-title":"Med. Phys."},{"key":"pmbab6e54bib019","doi-asserted-by":"publisher","first-page":"680","DOI":"10.1016\/j.ijrobp.2014.06.045","type":"journal-article","article-title":"Deformable image registration for adaptive radiation therapy of head and neck cancer: accuracy and precision in the presence of tumor changes","volume":"90","author":"Mencarelli","year":"2014","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"pmbab6e54bib020","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1120\/jacmp.v17i3.5735","type":"journal-article","article-title":"Benchmarking of five commercial deformable image registration algorithms for head and neck patients","volume":"17","author":"Pukala","year":"2016","journal-title":"J. Appl. Clin. Med. Phys."},{"key":"pmbab6e54bib021","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1186\/s13014-015-0372-1","type":"journal-article","article-title":"Validation of Varian\u2019s SmartAdapt deformable image registration algorithm for clinical application","volume":"10","author":"Ramadaan","year":"2015","journal-title":"Radiat. Oncol."},{"key":"pmbab6e54bib022","doi-asserted-by":"publisher","first-page":"712","DOI":"10.1109\/42.796284","type":"journal-article","article-title":"Nonrigid registration using free-form deformations: application to breast MR images","volume":"18","author":"Rueckert","year":"1999","journal-title":"IEEE Trans. Med. Imaging"},{"key":"pmbab6e54bib023","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1016\/S1361-8415(98)80022-4","type":"journal-article","article-title":"Image matching as a diffusion process: an analogy with Maxwell\u2019s demons","volume":"2","author":"Thirion","year":"1998","journal-title":"Med. Image Anal."},{"key":"pmbab6e54bib024","doi-asserted-by":"publisher","first-page":"2887","DOI":"10.1088\/0031-9155\/50\/12\/011","type":"journal-article","article-title":"Validation of an accelerated \u2018demons\u2019 algorithm for deformable image registration in radiation therapy","volume":"50","author":"Wang","year":"2005","journal-title":"Phys. Med. Biol."},{"key":"pmbab6e54bib025","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1002\/acm2.12437","type":"journal-article","article-title":"Framework for the quantitative assessment of adaptive radiation therapy protocols","volume":"19","author":"Weppler","year":"2018","journal-title":"J. Appl. Clin. Med. Phys."}],"container-title":["Physics in Medicine &amp; Biology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ab6e54","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ab6e54\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ab6e54\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ab6e54","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ab6e54","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ab6e54\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ab6e54\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ab6e54\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ab6e54\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ab6e54\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ab6e54","content-type":"text\/html","content-version":"vor","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ab6e54\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,13]],"date-time":"2025-12-13T14:29:00Z","timestamp":1765636140000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ab6e54"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,1]]},"references-count":25,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2020,3,6]]},"published-print":{"date-parts":[[2020,3,1]]}},"URL":"https:\/\/doi.org\/10.1088\/1361-6560\/ab6e54","relation":{},"ISSN":["0031-9155","1361-6560"],"issn-type":[{"type":"print","value":"0031-9155"},{"type":"electronic","value":"1361-6560"}],"subject":[],"published":{"date-parts":[[2020,3,1]]},"assertion":[{"value":"Data clustering to select clinically-relevant test cases for algorithm benchmarking and characterization","name":"article_title","label":"Article Title"},{"value":"Physics in Medicine & Biology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2020 Institute of Physics and Engineering in Medicine. All rights, including for text and data mining, AI training, and similar technologies, are reserved.","name":"copyright_information","label":"Copyright Information"},{"value":"2019-07-02","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2020-01-21","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2020-03-06","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}